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ABSTRACT 


This thesis reports the design, conducting, and results of an experi- 
_ ment intended to measure the paging rate of a virtual memory computer 
system as a function of paging memory size. This experiment, conducted on 
the Multics computer system at M.I.T., a large interactive computer utility 
serving an academic community, sought to predict paging rates for paging 
memory sizes larger than the existent memory at the time. A trace of all 
secondary memory references for two days was accumulated, and simulation 
techniques applicable to “stack” type paging algorithms (of which the 
least-recently-used discipline used by Multics is one) were applied to it. 
A technique for interfacing such an. experiment to an operative com- 
puter utility in such a way that adequate data can be gathered reliably 
and without degrading system performance is described. Issues of dynamic 
page deletion and creation are dealt. with, apparently for the first re- 
ported time. The succeasful Ber copennes 08 nts. xg 
viability of performing. this type of maaguxement . 
The results of the. SPREE, are Bite, 
paging behavior. 
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Introduction 


1. Brief Statement of the Problem 


In this thesis, we describe and report the results of an experiment 
designed to predict the performance of automatically managed mltilevel 


memory systems for a previously unexplored range of primary memory sizes. 


2. Summary of Result 
We have developed techniques for predicting memory system rer 


mance on an operative computer utility, ‘utilising an. ‘Siicmatically man- 
aged multilevel virtual memory. Based upon .eatabliahed theoretical tech- 
niques, we have developed techniques -to -extract,.the mecessaxy. data from 
a computer utility functioning under a live load, Jn doing so; we con- 
sidered: probleas of dynamic creation and deletion ‘Of pages which appa- 
rently have not been dealt with previously. .-The viebility of these tech- 


niques was demonstrated by performing several measurements. _ 


Using these techniques, we have found that, on the measured system, 
the rate of accesses to data outside of primary memory decreased drasti- 


8 bits (6 million 


cally as primary memory size is increased above 2 x 10 
36-bit words, or 24 megabytes). We have found that the mean time be- 

tween these accesses, as a function of primary memory size was best ap- 
proximated by a function of at least the second order, and possibly ex- 
ponential. Previous research on the system under consideration showed 
a linear function to hold for primary memory size up to 1.3 x 10° bits 
(4 million 36-bit words, or 16 megabytes) (S1). Although these results 


do not attempt to characterize Multics, we believe that they are rea- 


sonably representative of the observed class of user behavior. 


3. Summary of the Work of this Thesis 


By means of an experiment on the Multics computer system (B2), 
running on the Honeywell 645 at M.I.T., we have arrived at measurements 
of the predicted reference rates to..secondary memory for hypothetical 
extensions of primary memory. These measurements were made on an actual 
user load, the M.I.T. community, and. not any sort of benchmark or test 
‘ load. From these iganaeéaieate: models of program behavior in ILRU*~-man- 
aged storage hierarchies can be derived... We suggest here one such model. 

The essential technique for deriving these predictions from such 
. measurements is known in the literature :.(€1,C2) as the “extension:.prob- 
lem''. It is based upon the properties of a class of memory management 
algorithms known as "stack algorithms" (M1), which include IRU. Using 
these properties, we were able to simulate ‘the operation of the LRU al- 
gorithm for larger primary memory sizes than the actual one present for 
the identical user load. The input to this simulation was a history of 
all references to data outside of primary memory, specifically, on disk, 
during the period of measurement. It is a property of the stack al- 
gorithms that one measurement and simulation can be used to predict se- 
condary memory reference rates for all primary memory sizes. 

The work reported in this thesis is significant because it is both 
the first measurement of this type on a paged, segmented, multiprogrammed 


computer system which has been reported, and an extension of our range of 


*LRU, for Least Recently Used - a memory management policy whereby the 
least recently used data is moved to slower memory when space is needed 
in faster memory. 
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knowledge of the so-called "headway" function which we have described 
above. Previous measurements of this function (S1) involved other tech- 
niques, and only investigated it for primary memory sixes of up to 1.3 x 
10° bits. Our measurements explored regions approaching 4x 10° bits. 
Although there is no inherent limit on the range which could in principle 
be explored by our techniques, the limitation of our explorations is due 
only to the noteworthy fact that over a-day's runsing.of the experiment, 
no more than 4 x 10° bits. of infermation were referenced more than once 
by the M.1I.T. community. 

’ The significance of the actual resulting measurement is twofold: 


First, it provides an example of typical behavior for. the. measured sys- 


tem.. Second, it suggests more general models of: program: behavior. 


4. Structure of this Thesis 


Chapter 1 discusses the concepts of paging and. virtual memory. We 
provide justification for the types of: statistics and models we seek and. 
describe how to use them in performance: predicttons... We discuss previous 
research in this area, and provide a more detailed statement. of the 
novelty of this thesis. 

Chapter 2 describes the experiment. We describe the relevant fea- 
trues of the so-called "stack" algorithms (M1), and the. extension prob- 
lem. We discuss the problems of adapting this type of experiment to the 
mrltilevel memory system of Multics. We describe the difficulties in 
performing this experiment on an operating computer utility, and the 
solutions we adopt. 


Chapter 3 gives the results of the experiment. The results are pre- 
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sented graphically, and we suggest their interpretation. We analyze 
these results, and provide a detailed error analysis. 

Chapter 4 is a summary of the work done. We suggest future direéc- 
tions for research, and pose some of the questions left unanswered by 
this thesis. 

There are three appendices. 

Appendix A is an extremely detailed description of the Multics paging 
control algorithm, as it was at the time of the experiment. We desctibe 
it on several levels, allowing comprehension by the reader on whichever 
one he chooses. This background is useful for full comprehension of! cer- 
tain design decisions in the planning of the experiment. It is also!the © 
first publication of this algorithm at this level of detail (Corbaté' (C4) 
provides a less detailed discussion). 

Appendix B describes how the actual events of Multics memory mahage- 
ment were mapped into the idealized events of theoretical interest to the 
experiment. We describe the modifications and the interface to the Mul- 
tics supervisor necessary for this experiment. We assume that the reader 
has some comprehension of the previous appendix. 

Appendix C is a graphical presentation of user load, idle time, | and 
paging overhead on the Multics system on the days of the experiment.’ 
These figures were derived from routine metering performed by the adihin- 


istration of the M.1I.T. Information Processing Center. 
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Chapter 1 
1.1 Memory Performance Prediction as a Gal 


As digital computer systems have increased in size and complexity 
since their inception almost twenty years ago, so have the memory archi- 
tectures required to support -increasingly advanced applications and -sys- 
tems. What is more, progress in. memory, techaplogy has created,a plethora 
of memory media, ranging over a wide gamut of costs, speeds, and pro- 
perties. The desire for increased. throughput, .and- in real-time systems, ~ 
the desire for quick response, create a need. for: the fastest. memory tech- 
nology available. The fastest media, however, ate. almost always the most 
expensive on.a cost-per-bit basis. Thus, for a.given computer system. to . 
achieve or approach desired goals of memory. access. speed within a given. 
economic constraint, it becomes useful for memory. systems. consisting of 
varying amounts of mixed memory technologies to be used in one installa- 
tion. 

Most computers. of the past. twenty years have used magnetic core as 
their main, or primary memory. That is.to say,.the processor. was capable 
of fetching data end instructions, only from core.memory.. Further memory 
demands were met by the use of tapes, disks ,, and other bulk media, whose | 
contents could be transferred in or out of selected. areas of primary 
memory by explicit program request. Most.of the programs and operating 
systems designed for this type of architecture allocated these areas for 
input/output transfers in fixed, specific regions of primary memory. 
When programs could not fit in their entirety in primary memory, they 


were divided into independent pieces, or overlays, which were transferred 
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in and out of primary memory essentially at their own discretion. 

In the last few years, a strategy known as virtual memory has achieved 
popularity. With this scheme, programs are allowed, effectively, to re- 
ference data or instructions in primary memory or on any secondary storage 
device in an identical manner, creating the impression of a very large, 
or in some cases, conceptually infinite primary memory. References to 
secondary memory cause software intervention, signalled by specialized 
hardware, which results in selected code or data fragments being read into 
primary memory. Clearly, this implies replacement of gana Sthse code or 
data currently in primary memory, and in order to facilitate this task, 
such systems divide all primary and secondary storage into equal-sized 
areas, called blocks, or page frames. Information in the system is di- 
vided into pages, which may reside in various page frames at various times. 
This implementation of virtual memory is thus known as demand paging, as 
pages are read in on demand, i.e., when referenced. The selection of 
appropriate pages in primary memory for replacement is a critical issue, 
and is still a basis for much further study. 

A page fault, as the software-assisted fetch of a page not in pri- 
mary memory is called, represents lost time. The time required to ac- 
cess and transfer the copy of the page on secondary storage is time during 
which the requesting program may not run. The time that a processor must 
spend in page fault software, deciding on an appropriate page to replace, 
is a system overhead, which does not contribute to the progress of deed 
programs. Multiprogramming, a scheme almost universally used on medium 
and large scale systems, allows processors to serve one idee" program 


while another's is suspended, say for a page fault. But even here, most 
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systems limit the degree (number of simultaneously runnable users) of 
multiprogramming, aid page faults can lead to a situation where a pro- 
cessor apenas an undesirably large fraction of its time sitting idle, 
accomplishing no function at all. Furthermore, the primary memory space 
occupied by all of the faulting program is unusable by any program for the 
duration of the transfer. Thus, the minintsetion of page faults in a 
virtual memory system is extremely desirable. It ig an important function 
of the page-replacement algorithm, as the procedure which selects pages 
for replacement at page-fault time is known, to attempt to minimize the 
number of page faults in the forseeable future. ‘These decisions are usu- 
ally made with information gleaned from obsexvation o£ page usage in the 
immediate past, occasional knowledge of predicted page usage patterns, 

and some general models of program behavior. 

Many page-replacement algorithms have thus been designed for virtual 
memory systems with the explicit objective of minimizing page faults. 
These algorithms are subject to mathematical analysis, which is not true 
of arbitrary user programs. Hence, by odnceat observation of the storage 
references made by a program or multiprogrammed collection of programs 
(although the latter clearly requires some further remarks) we can ana- 
lyze its interaction with any given page-replacement algorithm running 
in any given size of primary memory, and ascertain which page faults 
would or would not have occured had primary memory been some other size. 
These techniques are not in general applicable to non-virtual memory sys- 
tems, for many programs have no idea of how large a memory they are 
running in, or how to take advantage of it, and thus explicitly-requested 


data transfers are not affected by changing memory size in any inter- 
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esting or easily analyzable way. 

The ability to determine page fault rates (page faults per unit time) 
for different memory sizes is a powerful tool in both performance analysis 
and memory system engineering. Sekino (S2) has shown the explicit depen- 
dence of response time and throughput in multiprogrammed systems on the 
mean headway between page faults (MHBPF). This quantity describes the 
Mean amount of useful work done by user programs between each two page 
faults. It is most conveniently measured in total references to the vir- 
tual memory. If the mean amount of system overhead associated with a 
page fault is known, as well as a proper characterization of system idle 
time,:we may compute MHBPF from the mean real time between page faults... 
(MIBPF) and the processor reference rate. Hence, predictive techniques 
to ébtata page fault rates for contemplated memory sizes can be used to 
deduce the system throughput and response time.figures. which would result. 
Hence, if one can indeed predict these figures, the economic tradeoffs 
involved in acquiring improved memory systém performance by increasing 
primary memory size may be evaluated more methodically. 

The use of more than one type of secondary memory in a single system 
results in a situation where the average time to access a data item in 
any part of the storage system is a function of both the average access 
time to a data item in each unit and the probability of accessing that 
unit. In a demand paging system, the probability of accessing each unit 
is the sum of the probabilities of accessing each page stored on it. If 
one can associate these probabilities with given pages of such a system, 
one can create a composite memory system with an optimal average access 


time within any given cost constraint. Ramamoorthy and Chandy (R1) have 


Eo: St Rap Set 
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given an algorithm, whereby such a system may be constructed out of any 
collection of memory types, whose speed and cost-per-bit characteristics 
are known. In any case, it is clear that one should keep the pages with 
the highest reference probability on the fastest storage.devices. Al- 

though. the identities of these pages may. he determined by experimentation, 
observation, and program analysis, one can view these. probabilities and/or 


identities as functions of time... Thus, ane.can devise.algorithms which 


attempt to maintain pages with given ranges -of :next-reference probabilities 


on appropriate storage devices. It should. be fairly apparent that this 
problem is identical to that of maintaining. pages in primary memory with 
the intent of minimizing page faults. This wili. be:-discussed more. later 
on. Thus, the design of an optimal multilevel storage. system, as such 
configurations are known, can also be analysed by the. techniques :of pri- 
mary memory paging analysis. Again, the asgumption of an appropriate . 
model of. program behavior, both in general and.. for the particular -system., 


at hand, is of crucial importance. 
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1.2 Program Reference Patterns and Models 

Computer programs being among the most deterministic of all things, 
any characterization of the data reference patterns of any particular pro- 
gram may be obtained by the simulated running of that program and the 
observation of whatever characterizations are desired. However, system 
engineering requires characterizations of programs which are to be run, 
which, for the most part, have not yet been written. In a given computer 
system, running under a given operation system, most running programs 
have many features of their memory usage patterns in common. For instance, 
in an operating system where an Algol-60 or PL/I-like run-time stack is 
native to the environment, the pages containing the top of the stack will 
always have a higher next-reference probability then page representing 
lower regions. If the supervisor itself is paged, i.e., running in a 
virtual memory, the same as users' programs, the supervisor has its own 
reference patterns which will be present in any run of the system. The 
same is true of compilers, assemblers, system utilities, library routines, 
and other service programs. Code generated by the same compiler is likely 
to produce certain common features in its reference patters, particularly 
on a local level. Thus, there is great value in observing typical be- 
havior of programs in a large computer system, and trying to formulate 
some model which is in some sense average or typical. 

In a multiprogrammed system, this averaging is done for us in real 
time. An experimental observation of program behavior in a multipro- 
grammed computer system, made over some reasonable period of time, say a 
day, will produce a characterization of typical system behavior, if one 


indeed believes that such exists. This characterization takes into con- 
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sideration all of the programs run in that day, and relies on the common 
features of programs discussed above to have any validity at all. The 
interval of a day is chosen as reasonable, for that is the cycle time of © 
many forms of human interaction with a. computer. People deal with an 
interactive computer system for several days, doing the same type of work 
at similar hours in the day. 

The particular model of reference behayior that we seek describes 
next-reference probability of pages in a virtual memory system as a func- | 
tion of position in a certain dynamic ordering, known as a stack, imposed. 
by the page-replacement ‘algorithm. The class of algorithms amenable to 
this analysis are precisely those which would keep the top n pages of this 
ordering in an n-page primary memory, were it. used.te manage such... This 
will be discussed more fully in section 2.1. What is important here is 
that we can arrive at a function p(x), where p.is the. probability of re- 
ference to. position x in this ordering. It is the object.of that sub-class 
of these page-replacement algorithms which are actually useful for memory 
management to make this: function monotomicaily deczgesing. If.the al-. . 
gorithm actually succeeds at this, it is clear that then pages which are 
most likely to be referenced will probably be in the Repage primary memory, 
and thus, the page-replacement algorithm has succeeded in minimizing re- 
ferences outside of the! n-page primary memory, or- page faults. In the 
case of miitilevel memories, we can pick out. whatever positions in the 
ordering are appropriate, by Ramamoorthy and Chendy's algorithm, and as- | 
sign them to whatever storage unit is required. C. K. Chow (C3) has also 
given an algorithm where an optimal multilevel memory system within a cost 


constraint may be constructed directly from the function p(x). 


19 


1.3 The Experimental Determination of Predicted Headways 


If we accept the function p(x) as a waita characterization of average 
and typical behavior in a multiprogrammed system, we may predict page 
fault headways for hypothetical memory extensions from it. Furthermore, 
the function p(x) may be measured experimentally. In this section, we show 
how to approximate and use p(x) in this way. 

x is the position of a page in the algorithm-imposed ordering we have 
been discussing. Assume we have constructed the necessary tools to mea- 
sure r(x), where r(x) is the number of times a page in position x of the 
ordering was referenced. Assuming pages which were never touched to be in 
position "infinity" of the ordering, then the relative frequency of 


touching a page in position x is 


£(x) = r(x) 
¥ r(t) . (1) 


t= 1 
Here, the numerator is the count of references to position x, and the 
denominator is the total number of references to all positions. If p(x) 
is indeed a valid characterization, f(x) should approximate p(x). 

We have stated, that for the class of algorithms under consideration, 
the first k positions of this ordering at any time contain precisely those 
pages which would be in a primary memory of size k. Hence, references to 
pages in the first k positions of the ordering never cause a page fault 
in a k-page primary memory, and references to pages in any position beyond 
k always cause -page faults. Hence, if a program makes H references to 
the virtual memory, the number of page faults it will take in the course 


of those references is identically the total number of references made 
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to positions in the ordering beyond the primary memory size. Thus, the 


program, running in a k-page primary memory, will produce a mean headway 


MHBPF (k) = H 
5 r(t) (2) 


t=k+1 | 
This relation holds true for any primary ery. size k. If we have an 
actual system, running in a pelea ius memory of size n, we can predict the 
MHBPF which would result on this system were memory estacded to size E, 
E being greater than n. We assume that we can eecinive MHBPF(n) on the 
existing system, and that a tool for measuring r(t), for sig is available. 
Then the Same program which takes H virtual memory references will have a 
MHBPF in the E page memory of o's 


MHBPF (E) = | ne 
B r(t) (3) 


t= E+1 


We now divide equation (3) by equation (2), obtaining 


, (t) 
MHBPF(E) = _t = ntl * 
MBBPF (n) g i 
5 age EG) Co) 


Observe that this equation allows us to predict MHBPF from 4 mea~ 
sured MHBPF and measured reference counts, a fact which will be used 
later. We now rewrite equation (1) to read 

r(t) = £(t) E r(u) | (5) 
us 1 


letting t be what was x and u be what was t. Substituting G) in (4), 


eet 
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replacing r(t), we obtain 


f(t) = ru) ry 
u= lL 
MHBPF (E) = a ae es! 
MHBPF (n) a 7 
f(t) = r(u) >: 
u= 1 u= 1 
t = E+l 
y £(t) 
= t= ntl 
fee] 
> £(t) 
t = E+l 


Multiplying both sides by MHBPF(n), we obtain 


sy £(t) 
MHBPF(E) = MHBPF(n) t = ntl 
FR g(t) 
t = Etl 


(6) 


(7) 


This equation states that mean headway between page faults which would 


result from a memory extension to E pages may be computed from the mea- 


sured mean headway between page faults on the unextended memory, and a 


factor which is a function only of the program or programs being run and 


the memory sizes concerned. The work of our thesis is to compute this 


factor. 


vie Aegan Setatoes mb ae tps ie a late at aR eat ee ett coe sire cing, Oe Weekes ele 
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1.4 Previous Work in this Area 

Since the advent of virtual memory computer systems, the function 
MHBPF (x) has been of great interest, being an easily identifiable charac- 
terization of memory system performance. Investigators. have run many pro- 
grams in simulation, obtaining this mean headway as a function of memory 
size experimentally. Almost all of these experiments have been done on 
machines which attempt to 'compress' a program into a smaller space than 
that in which it was intended to run. Such systems may typically attempt 
to fit five or ten programs, each running in a 32 k virtual memory into a 
core memory of 96 to 150 k. In such instances, the set of pages referenced 
by each program is small, as is the soredetas set which it can reference. 
These sets of pages are usually disjoint, as they represent disjoint 
virtual memories. Virtual memory in his Caee is Sines technique to 
force several programs into a primary memory too small to contain all of 
them. 

Such work has been ew iuceeaty Belady (81), Belady and Kuehner (B3), 
and Fine et al. (Fl), among others. A large amount of this work was done 
on an IBM M44/44X, a 7040 type machine at IBM Research Labs adapted. to: 
demand paging. Belady and Kuehner report..an expected HBPF for single 
programs running on this system of the general form e = a an n being 
primary memory size. 

Brawn and Gustavson. (B4) performed some measurements of typical com- 
putational programs running on the same M44/44X. These measurements were 
significant as they are apparently the first reported measurements of 


programs specifically written for a virtual memory. They observed the 
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running time of programs, including page fault overhead, as a function 


of real memory size. No analytic models were suggested. 

Some performance analysis done by Schwartz (S3) on a Burroughs model 
6700 is also of interest Hanes In this system, all data available to a 
program is referenced as variable-size segments, brought into core on a 
demand basis. Program code and certain data segments are shared, and the 
amount of information potentially accessible to a program is extremely 
large. He reported headway functions of the form e = exp (an), variables 
the same as above, for missing-segment exceptions as memory size was 
varied. (These were actual measurements performed on various memory con- 
figurations.) 

The research which directly led to this thesis was done by Saltzer, 
and later by Saltzer, Webber, and Snyder(S1). Saltzer measured the MHBPF 
on the Multics system (B2) at M.1I.T., with two different sizes of configured 
memory. He obtained the result e = a*n, which has since been called the 
‘linear paging model’. Saltzer later reported the results of an experiment 
designed and conducted by Webber and Snyder, in which the reorderings of the 
list by which the Multics paging drum is maintained were observed. Using 
the techniques described in 1.2 above, MHBPF(n) was extrapolated to a memory 
size of 4000 pages (each Multics page is 1024 words by 36 bits), and was 


found to be still within experimental error of the linear paging model. 
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1.5 Novelty of the Work in this Thesis 
The work performed in this thesis was originally conceived as an 


extension to Saltzer and Webber's experiment, elucidating the nature of 
MHBPF(n) for n greater than 4000 pages. The limitations of the linear 
model weré sought, as was the nature of whatever model held beyond that 


range. 


This series of experiments on the Multics system is unique for several 


reasons. The data accessible to any program in Miltics is potentially 

the entire storage system, and all data accessés ‘are wade via the virtual’ 
memory mechanism. This is similar to the turrougha scheme, but dissimilar 
to the paged ‘compressing’ type systems described above. Furthermore, 
sharing is an extremely important consideration in Multics, as all pro- 
gram code, including thé supervisor, is shared. ‘This thesis is also 
‘apparently ‘the first reported attempt to deal with dynamically variable 
virtual memories, i.e., those whose size grows and shrinks on a second-to- 
second basis. The issues of dynamic page creation and destruction which 
result from this policy are systematically deate with ‘by our experiment. 


The use of virtual memory seems to be gaining in popularity as large 


general-purpose information systems become more comion. Increased interest 


in systematic protection schemes has resulted in many new designs for 
systems having segmentéd addressing features similar to those found in 
Multics. Demand paging has achieved considerably more popularity and 
widespread use than the Burroughs techniques as an implementation for 
segmentation, and has recently been added by IBM to their extremely popu- 


lar System/370. 
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For these reasons, we feel that experiments made on a Multics-like 
system are relevant to data systems in the near future, and the reference 
patterns observed may have some features which are in some sense 


characteristic of programs running in segmented, paged, environments. 
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Chapter 2 


2.1 Stack Algorithms and the Extension Problem 
The substance of the experiment pee totaed was to reconstruct the en- 


tire history of a day's LRU-maintenance of the Multics storage hierarchy, 
and attempt to predict page-fault headways for hypothetical memory con- 
figurations from this history. 

The basic strategy of memory simulation used was that proposed by 
Mattson et al. (M1). This technique, known as stack simulation, relies 
on the fact that a large number of useful paging algorithms, including 
IRU, have the property that after any fixed number of addresses in an ad- 
dress trace have been processed by the algorithm, ‘the pages which are left 
in primary memory are always a subset of what they would have been at the 
same point in the trace had primary memory been larger. This feature, 
known as the "inclusion property", thus defines the class of "stack 
algorithms". From this property, at any given point in the processing of 
an address trace an ordering can be constructed, The first page in this 
ordering would be that page which would then be in primary memory were it 
of single-page capacity, the second would be that page which would also 
be in primary memory were it of two-page size, the third that which would 
be added were memory of Gixcspege-slew: and so forth. The history of 
the processing of an address trace can be viewed as a series of these 
orderings, which are known as "stacks", the single page corresponding to 
unit-size memory normally being considered the "top". As each new re- 
ference is processed, the algorithm causes the stack to be reordered, 


possibly corresponding to page motion for some size memory. The top n 
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pages on the stack being the pages which would be in a memory of n-page 
capacity, any motion of a page into the top n-pages implies a physical 
reading of a page into primary memory. For a demand stack algorithm, this 
movement can occur only as the result of a page fault. Thus, we may infer 
the behavior of an n-page primary memory by observing the number of times 
that reference is made to position ntl or. beyond. in such a stack. As we 
have defined this stack and the class of algorithms processing it as main- 
taining the first n pages in this stack in an n-page memory, no reference 
to position n or below can ever cause a page fauit. Mattson's technique 
consists of ere a recorded or proposed address trace, running it 
through a program which constructs the sequence of stacks*just described, 
‘and actumulates the total munber of referentes.to each position therein. 
When the: processing of the trace begins, the stack is void, corresponding 
to’an empty primary memory. At least until a given page is fetched: into 
primary memory the first time, it will not have been in the stack at all, 
and its first fetch may be considered to have been made from position 
“infinity”. “As the trace progresses, and repeated references to pages 

are made, we accumulate counts for each position in the stack of how many 
times a page in that position was moved upward by the algorithm. It can 
be shown that for a demand stack algorithm, the only condition on which 

a page may move upward in the stack is that it is that page which has 

just been referenced. Simply, were this not the case, a page in position 
n would move into an n-1 page primary memory without having been refer- 
enced, and the algorithm would not be a demand paging algorithm. As the 
‘deb tation of the address trace, we can, for any n, sum the reference 


counts for positions ntl to the total final length of the stack, plus the 
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count for position “infinity, and this will be the mumber of page faults 
which would have transpired had that address trace been managed by the 
algorithm used in an n-page primary memory. Note that a single processing 
_ of the trace can be used to produce a result which can then be used to 
analyze any ‘ivpoebacicel-eimies size. 

. This.technique allows us to..ascertain-the page, fault count for the. 
interval under consideration for any contemplated memory size. By simply 
dividing the total system headway during. the actua].trace by this page- 
fault count, we may thus ascertain the predicted megn time between page 
faults (MHBPF). for that storage system, Furthermore, we.can plot the re- 
ference counts pivnacte pontcion, nowmaljzed with yespect._to the total 
number..of.reference counts, versus. the position number, and obtain a graph 
which describes what we shall call. access frequencies. With this, we. an 
analyze the behavior of multilevel memory systems processing this trace, 
and obtain an. optimal such ayécen within: cost. constraints as. described in 
Chapter 1. The shape of this graph also tells.us. mich about the relative 
success of the particular algorithm in managing that particular, address 
_trace, without regard to any single memory. configuration. We will con- 
sider the- particular graph in.the case of our cisuite in greater detail 
in the next chapter, and in so doing further consider such graphs in 
general. | 

Our experiment sought to learn the shape. and nature of this graph at 
positions corresponding.to memory sizes of many thousands of pages. In 
order to record a. reference to position n in a stack as described, there 
must clearly be n-1 items above it. This implies. that at least n dis- 


tinct items have been referenced by the time a reference to the a posi- 
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tion occurs. It can be seen that the extent of the address trace required 
to produce meaningful statistics at the ten-thousandth page position would 
require a prodigious address trace. At this point advantage may be taken 
of another remarkable property of stack algorithms. It is possible to con- 
struct the portion of the stack from position n+l to the end without a full 
address trace -- we use information extracted from a running algorithm 
managing an n-page primary memory at times when page faults occur. This 
is known as the "extension problem'' (C1,C2). The technique is as follows: 
we maintain the stack (the "extension stack") for positions ntl and be- 
yond. When a page fault occurs, we know that the page faulted on cannot 
be in the first n positions of the stack -- if so, it would not have been 
faulted on. We locate the page in the extension stack; if not there, we 
may consider it as having been at position "infinity". The counter cor- 
responding to the position from which the page was fetched is incremented. 
We remove the page in question from the extension stack: it is now in the 
top position of the real stack, which we are not maintaining. We now use 
whatever information is necessary, from that normally obtainable to the 
running algorithm, plus that we are maintaining, to reorder the extension 
stack according to the policy of the running algorithm. This reordering 
will usually include placing some page removed from primary memory by 

the running algorithm at some point in the extension stack. In the case 
where the replacement algorithm is LRU, the page removed from primary 
memory is placed on top of the extension stack, and all pages previously 
in the extension stack move down one location. Note that pages which 

were below the fetched page in the extension stack stay in place during 


the entire transaction. 
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The advantages of isles ‘a trace of a ruming algorithm in a large sys- 
tem measured over an extended period of time, ‘as opposed to: a trace ob- 
tained by simulation of a given program over @ necessarily much. shorter 
period of time are straightforward. We are interested in system perfor- 
mance on an hour-to-hour, not second-to-second,. basis, and day-long mea~- 
surements of a live system correspond to both the ‘time scale and load mix 
of interest. As long as the accuracy of the measurement can be maintained, 
this day-long extension measurement is much more. useful than the simulated 
running of a program. 

As a demonstration of the power‘of this extension technique, we may 
consider the Multics system: 400,000 references ‘to the virtual memory: oc- 
cur every second. Approximately 100 page faults. occur each second. Re~ 
cording 2 date items for each page fault, we have reduced the amount 
of data which must be recorded by. a factor “of two ‘thousand. 

It should be clear that ‘the results ‘of..this experiment, although simu~ 
lating hypothetical memory system performance,. do: ‘not: represent simlated 
results, The measurements made correspond .te an uncontrolled user popu- 
lation during normal working days, using arbitrary programs under 6-way 
muitiprogranmming. The results thus show Low a iypothetical memory sys- 


tem would have behaved under this ‘real user: load. 
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2.2 The Extension Problem and Multics 

The Multics system has a physical memory consisting of 256K (1 K= 
1024 36-bit words = 3.7 x 104 bits) to 384 K words (1.2 x 10” bits) of 
core, a 2000 to 4000 K word (1.5 x 10° bits) drum, and approxiastedy 
90,000 K words (3.3 x 10° bits) of disk, both moving and. fixed-head. The 
variabilities stated above are dependent. upon the time of day and the. user 
load, governed by administrative policy. The entire storage system is 
divided into 1024-word pages, and is managed by the demand paging mechanism 
(with the exception of several thousand words of non-pasgeable code and 
data, such as the code for the paging mechanism itself, which must be 
non-pageable in any case, and are thus net really of interest in memory 
performance prediction). The algorithm used to manage replacement of 
pages in the core memory is essentially IRU. The variation. from LRU is 
explained in etait in Appendix A. Essentially, within the constraints 
of operating system overhead and the precision of measurement of recency 
of use provided by the hardware, it tries to implement LRU as closely as 
possible. Aliso, a non-demand prepage/postpurge policy was in effect 
during these measurements, which caused some: pages: to. move in. and out of 
core outside of the control of the LRU algorithm. . 

The 4000-page paging drum was at this time being used in a mode 
which attempted to overcome rotational latency by making multiple copies 
(S4), in this case two, and hence was of 2000 page capacity during these. 
experiments. Since January, 1972, the drum has been used as part of a 
hterarchically managed storage system, as a buffer. between core and the 
disk storage subsystem. In such systems, one attempts to keep pages 


with the highest access frequencies on the fastest devices, in order to 
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minimize the system's mean access time. In an. automatically managed sys- 
tem, the identity of those pages is constantly changing. As the stack 
access frequency graph discussed above can be used to assaciate access 
frequency with stack position, page replacement algoritims identical to 
those used to manage primary memory are frequently used to manage other 
devices in a hierarchical memory system te. achieve ‘this end, In the Mul- 
tics system, another near-LRU algorithm. is used to manage the drum, which 
is described as well in the Appendix. The drum menagement algorithm at- 
tempts to maintain copies of the top 2000:'pages of the theoretical stack 
corresponding to the LRU algorithm on the:drum. Yhe model. of. program be- 
havior implied by the IRU algorithm, and verifded by the results. of this 
experiment, implies that.these pages are the most likely to be referenced, 
and at the time they are on the drum, thus have the highest access fre- 
quencies. | 

As currently implemented, a page which has been faulted on, and is 
not on the drum, is read into core from the disk. It will not be written 
to the drum until the core management algorithm. decides to oust it from. 
core. This implies that the pages corresponding to that portion of the 
LRU stack representing core are not compketely a subset. of those on. the 
drum. Hence the drum will contain pages representing .a 2000-page con- 
tiguous portion of the stack, whose topmost extreme is. anywhere between 
the top of the stack and the size of core below it. Of the 256 to 384K, 
about 100K is not used for paging, leaving 150 te 280K for paging, thus . 
this variability represents about 7 to 15% of the size of the drum. 

The stack-reordering procedure of the IRU algorithm is one of the 


simplest possible: the referenced page moves to the -top position of the 
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stack, as is necessary in any demand stack algorithm. The pages between 
the top page and the old position of the referenced page all move down one 
position. Thus, to use the extension technique described above for the 
LRU algorithm, the only reordering information one need record is the 
identity of the page thrown out of position p (p being the size of the non- 
extended memory, in pages), which will then occupy position ptl, at each 
page fault, in addition to the identity of the page faulted on. The 
"pushed"' page becomes the top of the extension stack, the page previously 
there becomes #2, etcetera, all the way down to the former position of the 
faulted-on page. This is what we have done with the Multics core-drum 
combination, considering it as a 2000+X page buffer, where X is some frac- 
tion of the size of core, itself at most fifteen percent of the drum, to 
account for the top-of-drum variability described. As positions p+tl and 
on, in our case, correspond to the disk subsystems, we need only record 
disk reads instead of page faults. (It is instructive to note that within 
the entire operation of a Multics system, not a single direct-access I/0 
transfer is done outside the paging mechanism, pre-paging being included 
in this consideration.) As disk reads are two to five per second, we 

have thus reduced our data-gathering chore by at least 95%, The experi- 
ment of Saltzer, Webber, and Snyder, which was similar in intent to this 
experiment, but more limited in scope, has already produced results (S1) 
for primary memory sizes up to the maximum size of the drum. For this 
reason, we did not consider it worthwhile to attempt to gather data for 
that portion of the stack corresponding to regions in the drum. Hence, 
the application of the extension technique to this core-drum combination 


was adequate. All else that was needed was the recording of information 
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provided by the drum management algorithm as to the identity of pages 
thrown off the drum. It can be seen that these are the pages thrown out 
of the core-drum combination, given that. ro page is ever thrown out of 
core without having been written to the dru. 

In order to determine the validity of this technique, the necessary 
programs were written and tested on a stand-along Multics machine. This 
machine had 131,072 words of core and a 256~page.drum. Hence, most.of the © 
range of the regular Multics drum was in the extension region of this ex- 
periment. The mean headway curve resulting was very well approximated by 
a straight line, suggesting the linear paging model. This provided. a 
good deal of confidence in both the technique and the software. 

Hence, we mae two.types of motion between core-drum and disk. The 
reading of a page constitutes motion from disk inte core-drum. The 
writing of a page, however, does not constitute outward. motion. In gene- 
ral, writing is performed only when a copy of a page on disk is different 
from a drum or core copy, The outward motion corresponding to a read is 
really the claiming of the core or drum frame previously occupied by the 
page of interest. We call this phenomenom an "ousting". 

Unfortunately, a problem arises with even this simple model. Certain 
pages of the storage system, 3 in all, corresponding to the system's top- 
level directory, are special-cased by the paging. and drum-management al- 
gorithms such that they may never go on the drum. This is due to certain 
integrity issues involving the reliability of the drum and the extreme. 
difficulty in reconstructing the contents of this directory. Hence, 
these pages are eee written to the drum, and leave the "core" portion 


of the increasingly less theoretical LRU stack directly for the disk por-. 
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tion. (In this case, writing never takes place unless the concerned page 
has actually been modified (see Appendix A)). More unfortunately, these 
are among the most popular pages on the disk, as by the dictates of the 
LRU algorithm, they should have by all means been on the drum. Thus, we 
must check for pages being ousted directly from core to disk, and we thus 
have two varieties (core and drum) of ousting to be recorded. The inter- 
pretation of the data resulting from movement of these pages will be de- 
ferred until Chapter 3. 

Thus, we need record upward stack movement into the core-drum com- 
bination, meaning disk reads, and downward movement, meaning oustings. 
Another event of interest is the creation and deletion of pages. In the 
daeueat duplemsaterion of Multics, logical pages are created out of the 
void when a never before referenced page is referenced. By definition, 
all such pages contain zeros, and hence never involve disk reading. Fur- 
thermore, these page faults will occur regardless of what size primary 
memory is, and are thus not of interest in memory performance prediction. 
This last statement is somewhat subject to current design and user beha- 
vior. Were there a tremendous amount of fast, cheap primary memory, it 
is altogether possible that users would rarely delete programs or data, 
but simply rewrite or modify them, thus making page creation a much rarer 
event. We choose to ignore this possibility. 

In the following discussion, 'n'' represents the size of “primary 
memory’, in pages, in terms of the extension problem. In terms of the 
specific experiment on Multics, n is the size of the core-drum subsystem 
in pages. As was explained earlier, this is the size of the drum (2000 


pages) plus a fraction of the size of core. 
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Page faults which cause creation of pages involve neither disk traffic, 
idle time, nor mltiprogramming, and are thus not of interest in MTBPF cal- 
culations. Since all new pages are created in .this way, we find them natu- 
rally falling past position n of the complete..IRU stack, into the extension 
stack after sufficiently long disuse. Page deletion, on the other hand, 
can occur at any time in the life of a page. If a page is destroyed. so 
soon after its creation that it has never passed position n in the stack, 
we are oblivious to its entire existence. If, however, itis destroyed. 
at such time that it is beyond. position n, its deatruction must be accom- 
panied by its excision from the extension stack. -When a. page is destroyed 
in core. or on the drum, the next page to be £quited-on replaces it without | 
any page being pushed down the LRU stack. However, -the- position in the 
stack of the destroyed page is assumed by the page directly under it. The 
page fault following a page destruction creates only upward stack motion -- 
nothing is pushed down. 

Consider, in our theoretical n-page primary memory system, a page in 
position n of the LRU stack. This page is now destroyed... A page in posi- 
tion mim is now faulted on. In an actual memory system, this page will 
now be read into primary memory without any page being replaced,. the des- 
troyed. page having created an empty page frame, but the newly faulted-on 
page will be at the top of the IRU stack. The-formerly first to n-lst 
pages now become the second to nth pages in the stack. The ntlst (first 
page not in core) to mm-Ilst pages retain their original stack position. 
(See figure 2.1). The nim'th position in the new stack is in a situa- 
tion akin to that of the nth position after the deletion of the page 


there: the page in the nimtlst position cannot come up to fill the void -- 
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that would not be demand paging. Until some reference is made to a further 
position, say mmtk, this will be the case. At the time the nimtk refer- 
ence is made, position ntmtk now becomes anomalous. Hence, we create an 
anomaly when a page is deleted, which propagates down the stack as any re- 
ference is made beyond it. 

The strategy that we have chosen to deal with this, in the. simulation, 
is simply to excise a page from the stack when it is deleted. Thus, any 
reference to a position beyond the excision will be tallied as a reference 
to position x instead of x+tl. Note, however, that the position of the 
excision has then moved to x. All references in front of excisions are 
tallied correctly. The analysis of the inaccuracies resulting from this 
treatment is quite involved, and is covered in detail in section 3.4.3. 

Thus, the data items which mist be recorded in a trace are those re- 
presenting 1) feadinn of pages into memory, by demand or prepaging, from 
disk, 2) claiming of pages by ousting pages fae drum to disk or from 
core to disk, and 3) the deletion of pages from the storage system. Of 
these events, types 1 and 3 represent excision of a page from the exten- 
sion stack, while type 2 represents the pushing of a page on to the top 
of the extension stack. Events (1) also cayse the noting of the stack | 
position of the page read, and the incrementing of a counter corresponding 
to that position. There are actually some other events which must be re- 
corded in the case of the Multics system, but these are due to the parti- 
cular implementation of the core and drum management algorithms, and are 
discussed in Appendix Be 

The handling of page reads of pages which cannot be found in the 


stack, i.e., their first reference, requires some thought as to inter- 
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pretation. Were this experiment run for a sufficiently long time, the 
appearance of such pages would cease. Pages which are. created come down 
on the top of the stack, and any existent page which has not been refer- 
enced since the experiment began enters the core-drum. extension-stack com- 
bination once and never leaves, until it is destroyed. These first refer- 
ences, as discussed, are counted in the "infinity" position of the stack. 
These fetches of pages not in the stack accounted for roughly a tenth 
(7881/74530) of all disk page fetches. These. references do not affect the 
relative number of fetches to any two extension stack positions, as they 
would not be in a core-drum memory of any size until the first time they 
were referenced. Thus, when one considers disk accesses, one should con- 
sider these reads to be disk references in a core=drum. system of any size. 
However, the longer the experiment runs, the fewer will these references 
become. Thus, since we are interested in steady-state behavior, we have 
chosen to consider these reads a start-up transient, and not count them 
in any caiculation. They tell only of the length of the experiment, not 


of what is being measured. 
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2.3_ Performing an Experiment:on Multics 
Having developed the theoretical bases of the extension problem, and. 


adapted it to Multics, the next step was to: proceed. to: construct the 
necessary software to develop the extension address trace, collect it, 
and perform the LRU stack simulation with it..: 

A privileged-access facility was set up in the Multics hardcore super- 
visor specifically for this experiment. When enabled, a trace of all of 
the events mentioned. above was accumulated in a circular 1008-word buffer. 
Each trace item included the physical device address of some. page being 
read, ousted from core-drum, or deleted, infermation as to which of these 
events is represented, and a flag indicating, fer statistical purposes, 
whether or not it was. one of the previously. mentioned pages. which are aot 
allowed to go on the drum. Also recorded was iaformation allowing the 
program which inspects this buffer to syrchronize itself with it cor- 
rectly. <A program was developed which. inspected: this buffer regularly -- 
from the Multics standpoint, a privileged operation. This program as- 
sembled the buffer images into a continuous trace, which could be as. long 
as necessary, suitable for further, repeated processing. 

This strategy was decided upon because of the extensive time required 
to search an LRU stack for a given page, and the large amount of space re- 
quired to store this stack. This ruled out the possibility of having a 
special-purpose module of the Multics supervisor perform the experiment 
in real time. The performance degradation necessitated by the time re- 
quired to search and the space, which would have had to Beet non-pageable, 
to store the LRU stack would have been wholly unacceptable. Furthermore, 


the accumulating of the trace data for further processing allows many pro- 
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grams and versions of programs to be run on this data, increasing both its 
usefulness and the accuracy of the results obtained from it. 

One disadvantage of the data collection strategy described is the pos- 
sibility of the data collecting program losing synchronism with the circu- 
lar trace buffer, i.e., data being overwritten by new data be- 
fore it has been duly noted. This situation can come about when the data 
collecting program has made a decision about how often the buffer should 
be sampled, and an intense unexpected burst of activity causes the buffer 
to be written into significantly faster than before. The data-gathering 
program samples the buffer again, and notices that data has been lost, 
but anticipating further loss reschedules itself. Another way that data 
can be lost from buffer mis-synchronization is the data-gathering pro- 
cess falling behind in the multiprogramming queue due to Multics sched- 
uling policies and heavy user load. The implementation of the data- 
gathering program tried to compensate for this by being written as a multi- 
process program, i.e., a program running in a coordinated way in many pro- 
cesses at once. Not only did this give it a scheduling advantage, but in- 
creased the reliability of the data-gathering operation as a whole. 

Unfortunately, data losses of the types described were common, espe- 
cially in initial, developmental runs of this software. The greatest 
losses would typically occur at midnight, when a large number of user pro- 
grams scheduled to run then would, creating heavy paging activity refer- 
encing pages neither in drum nor core, and only one or two processes would 
be supporting the data-gathering operation. The extents and analysis of 
these losses are considered in the next chapter. 


A danger of running a large complex data-gathering system in many pro- 
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cesses is that of creating a great deal of activity which would bias the 
result of the measurements by measuring itself. The sharing features of 
. the Multics system helped counterbalance this effect: all of the data. 
bases and procedures of the. data-gathering system were fully shared, having 
only one copy. Only the per-process work axeas were not shared. : The 
actual data-gathering system, in order to handle the-control of multiple . 
processes, possibly multiple terminals, and dynamic scheduling was, in 
fact, quite complex, requiring six separate procedures. : The shared data — 
bases and procedures totalled ten pages. Approximately two pages of work 
area per process were needed. 

The data gathered was stored in data segments in the Multics virtual 
memory. The stack simulation was subsequently performed, using this data 
as the extension address trace, exactly as described above. The procedures 
which performed this reduction ran in an unrestricted Meltics environment, 

' and hence had practically -no restriction. on time or space. The IRU stack 
was represented as a list, in which each node represented a stack posi- 
tion. A push of a page ‘onto the top ofthe stack: required the allocation 
of a new'node, and the redefinition of this node :ag the top of the stack. 
This node was then made to point to the former stack top. The:.excision of 
a page from the stack required locating the node corresponding to this page 
{each node contained a physical page address), the reallocation of this 
node, and the recotinécting of the list around it.’ For trace data repre- . 
senting disk reads, however, it was necessary to: ascertain the position in 
the list of the relevant page. This required a search of the entire list. 
In order to reduce the work of discovering that a page was not in the list 


at all, a bit table was constructed, describing, for éach possible physical 
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disk address, whether or not it was in the list at all. This saved the 
necessity of walking the entire list. 

The above list-maintenance algorithm spends a great deal of time 
searching the list to determine the position of pages in it. Several al- 
gorithms were considered to avoid the seemingly crude strategy of linear 
search, but most of these algorithms caused the list to grow increasingly 
disorganized, requiring periodic time consuming re-organizations, or re- 
quired large amounts of data movement, a poor approach in a paged system. 
Because of the availability of a stand-alone machine which could easily 
provide the computer time necessary to perform this processing, the develop- 
ment of a better list-maintenance algorithm was not pursued further. 

The result of the stack simulation was a table, describing for each 
position in the extension stack, how many times a page in that position 
had been referenced. The sum of all of these counts, plus those at posi- 


tion ' 


‘infinity, represented the total number of all page fetches from 
disk during the period of the measurement. Although a graphical display 
of this information is of some interest, the calculation of MHBPF was the 
immediate objective. Thus, a table was created displaying, versus exten- 
sion stack position, the total number of fetches observed divided by the 
sum of the counts for all of the positions further down the stack. For 

a given position N, the interpretation of this number, x, is as follows: 
had memory €ore/drum) been extended N pages above its actual size, we 
would make one disk reference under that circumstance for every x refer- 
ences we make now. We thus refer to x as ‘references per exception'. 


Note that we have not included the “infinity'' fetches in the 'total re- 


ference’ count in the actual results shown here, for the reasons dis- 


cussed in section 2.2. 

One may choose to interpret x as the "relative increase in mean head- 
way", which is to say, the factor by which mean headway will increase. over 
its current value if the extension of N pages is made to core-drum. For 
instance, it "ectecencsa pet exception”: were 5 at N.=.4000 pages, the in- 
terpretation would be: If we added another 4000 pages. to the drum, we | 
would fault to the disk one-fifth as often as. we do now.. This consti- . 


tutes, in one sense, the raw data result .of .the experiment... Now, 


References observed ._______ » Mean time between measured references . 
Reterences beyond position N 


=. References observed sy nt 
References beyond position N i Riterences observed 


Time duration of experiment _ 
References byond position N 


Mean time between references beyond position N 


Expected mean time between references were memory extended by N 


Multiplying this number by the measured system headway in virtual 
memory references during the experiment, and dividing by the ‘time dura- 
tion of the eiperinene- we obtain the expected mean headway between page 
faults were memory extended by Ne 

We have displayed both references per exception, versus memory exten- 
sion size, and predicted inter-reference headway, a a function of memory 
size. | : | 


Note that in all of this discussion, mean time computed from an ex- 
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periment taking many hours must be taken quite literally. The averaging 
effect over a day of usage very ine peenees a peayy user load and solely 
the data-gathering program* produc ae: a “result which is really applicable 
to neither, but only a theoretical load somener® in between: For this 
reason, we feel that ‘references per exception’ is a more useful tates 


Pom FR 


pretation of the results of this experiment than ‘mean time between | ex- 
ceptions’ é AEbempt ing to tune a system to the theoretical ‘point described 
by such measurements will not het the system when it needs ae most help. 
The reference counts and references per exception were subsequently 
displayed in printed tabular form, and the rererenees pee exception ver~- 
sus stack position plotted on a = Ercebere Car tec 4020 Microfilm Recorder. 


Some of these results are reproduced and eiacuases: an detail in the next 


chapter. 


*Some more precise descriptions of the exact user load during the experi- 
ment are provided in Appendix C. 
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3.2 The Results of the Experiment 


The form that we have chosen to display is that of an "exception 
ratio", or MHBPF(E)/MHBPF(n), where n is the adjusted 'primary' memory 


size (core-drum) as explained in section 2.2, and E.is a hypothetical 


memory, both in pages. This exception ratio is the quantity expressed by 


equation 1.6. We plot this ratio versus primary memory extension in 
figure 3.1. We express the abscissa of our ack as- ‘memory extension’, 
which is the hypothetical increase of core-drum instead of absolute memory 
size because of the vaviaplive of the size of core-drum as discussed in 
Chapter 2. The size of core-drum is not the sum of sizes of core and 
drum, because of duplications, created pages in core which have not been 
copied out to drum, and possibly even different configured sizes of core. 
The extension size to core-drum is meaningful however, because the data 


and results derived from the measured data represent the behavior of a 


hypothetical extension of the given size, oblivious to all of the above 


considerations. If a figure for the size of core-drum is needed, 2100 
pages is reasonable. The shape of this graph suggests an exponential be- 
havior. Thus, we next plot this ratio on a logarithmic vertical axis, to 


better view this behavior. This is figure 3.2. The plot almost traverses 


the graph diagonally, suggesting the straight line which would correspond 
to an exponential. We have drawn a straight-line approximation, which 


“corresponds to 


(E-n)/(7.00 x 10’ bits) 


MHBPF (E)/MHBPF(n) = 3.42e (1) 


The surprising closeness of the dtm 21 and dtm 23 plots gives some con- 


fidence in this result. A similarity to the unpublished Burroughs re- 
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56 


sult (S3) mentioned in Chapter 1 may be noted, subject to the limitations — 
discussed there. This similarity goes only as far as that Schwartz no- 
ticed an exponential mean-headway function for missing data in peimaty 
“memory as memory size was increased. 

It can be .seen that the low regions, i.e., below 2000 pages above the 
drum, fall short of the expmearial approximation suggested. Thus, we pro- 
vide figure 3.3, in which we display the low region of figure 3.1. This 
| plot shows a decidedly lese than exponential behavior. From-one view- - 
peiat. this ta-coitonting: as the experiment of Saltzer, Webber, and Sny- — 
der (81) measured this function in this tange, and obtained a linear fanc- 
‘tion. However, our plot seems to grow comsiderably faster than linearly a4 

in this region. This can be seen as a besten A different change in be-. - 
havior. ok o 

. An attempt to resolve these contradictions as provided wy figure ake 
4m which both memory extension and exception. ratio. are plotted logarithn- .: 
ically. It is seen that the higher regions of this plot approach quad- . 
ratte. slope, and the increasing slope lends more ‘confidence to the ex- 
ponential suggested above. However, no unsfote quadratic curve suggests 
itself. | 

The most that can be said is that MHBPF(E)/MIAPF(n) is a function of 
at least second order, as.E-n exceeds 3000 pages. 

We also provide figure 3.5, in which we plot MEBPF(E) directly as a 
function of mecsory extension, by multiplying the ordinate of figure 3.1 by 
the measured MUBPF on. ee Se eeeneeeees: Rote -that- @- gives exception ratio: 
does not correspond. to the same MEBPF on both sapevemicita: as different 


mean headways were observed. This is due to the variability of the system 
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load on these days. The difference, however, is not very significant. 


We write p 


Avene eno oun a ‘eng ae 
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the polynomial headway function MHBPF(x) = ax” giving in general 


k 
(x) = kn” 
Py tl (12) 


which subsumes (8), (10), and (11). The exponential model (9) is the only 
one of these probability distributions which is characterized by an inde- 
pendent parameter, y. Letting \ = 1/y, we rewrite (9) as 


P(x) = se M/A (13) 


\ has the dimensions of pages. It in some sense characterizes a ‘radius 
of locality of reference’ of the programs running. It is the mean fetch 


depth into the extension stack. 
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3.4 Accuracy of the Reported Results 


The question of accuracy of the results of a supposedly deterministic 
simulation seems at first to be unnecessary. However, this simulation was 
based upon a measurement. Thus, the techniques used to interface this ex- 
periment to the Multics system (see Appendix B) became a source of inac- 
curacy. Furthermore, the behavior of anomalous pages (the so-called "glo- 
bal transparent paging device" pages) caused significant deviation from 
the assumed LRU model. The deletion of pages in LRU list created prob- 
lems, as an inordinate amount of effort would have been required to handle 
these correctly (see Chapter 2). 

Thus, we will consider three sources of inaccuracy: lost data, glo- 


bal transparent paging device pages, and list deletions. 


59 
3.4.1 The Effect of Lost Data 


The loss of data was due to failure to retrieve it from the Multics 
supervisor before it was overwritten. This was a consequence of the cir- 
cular buffer strategy chosen to solve the problem of real-time storage of 
this data. These strategies were discussed in detail in section 2.3. 

The effect of these losses are twofold: some counters for stack posi- 
tions were not incremented for lost data, and the ordering of the stack 


was affected by this lost data. We consider these problems separately. 
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3.4.1.1 Lost Counter Accuracy 


In order to deal with either of these problems, we assume that lost 
data has no correlation to page reference patterns. We thus deduce that 
it shares the same distribution over stack position as the successfully 
accumulated data, and the shape of the resulting histogram is not se- 
verely affected by this loss. For the measurement "dtm 23", the most suc- 
cessful and accurate of thos made, 435 trace items were lost out of a 
total of about 200,000 successfully recorded items. This represents a 
total inaccuracy in counting of less than one quarter of a percent. For 
the slightly less accurate "dtm 21", 1200 items were lost at various times. 
Measured against the 150,000 items successfully collected here, this is 


still less than one percent. 


a RBS Pe PTS ae 
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3.4.1.2 Stack Shifting Inaccuracies 


This type of inaccuracy, resulting from inaccurate Feconstruction of 
the LRU extension stack, is eoustdecablys more subtle, and dmagite in its 
effect. Failure to notice certain movement into aa out of the extension 
stack causes the stack to stray courosaivels artnet &ron s peuiatie re- 
construction. Items remain in the simulated extension stack which were 
in fact removed by lost data, and items which should pave peer pushed on 
its top are not so pushed. The result of tral not betng removed cor- 
rectly is twofold: first, the item will appear twice in the stack when it 
is pushed out legitimately onto the top of the extension stack in the fu- 
ture, and items further down the stack than the false appearance will have 
their stack position incorrectly Sedouded: The double epresr ance is only 
a problem because of the latter eetect the ptack menee te algorithm of 
the simulation program used a bit table to Fecore the pnowe Presence of 
every storage system page in the extension stack. Thus, ihe legitimate 
pushing (ihe eesons time) of such a page has no effect, and the later 
fetching of that. page from the Bcension stack fetches tha: covredt in- 
stance, and the bit table indicates the page as no longer being in the | 
stack. | | . 

The result of items not being mishied because of failure to ieeoed 
their pushing is similar. Their abyence at the pel of ‘the extension stack 
causes all items below then, which = ee entire acece to have their sents 
tions incorrectly recorded. aneee items, appear one position higher than 
they should be, for each missing item. ‘tvs, until ne oe item is 


later requested oo he stack, by tcbraher of a recorded fetch, all items 


cey oe 
ere, 4 


which were on the stack before the failure to peer e the miosis ites will 


Ne 
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have their positions incorrectly tallied. Kiso: a later reference to the 
unpushed page will be recorded’ as a transient, seein ob ack page , fetch, as 
discussed ‘in section 2.2. the effect of not account ing this fetch to any 
stack position was already discussed. : 

Note that the effect of pushing and then fetching any single page has 
no effect on the extension stack orderings before the pushing and after the 
fetching. That is to say, if the pushing ‘and fetching were not ‘recorded 


at all, eely the stack orderings between the two would be incorrect, Thus, 


if at any one time a data retrieval from the hardcore supervisor notices 


that x data’ items were lost, all such push-fetch paire within the X lost 
data items have no - stack-reordering inaccuracy aaeocieted with them, and 
only the lost-counter taaccurecy occurs. vA reference-push pair, on the 
other hand, causes both ‘types of ieee Although it seems. ‘evident, by 
locality of reference, that any string of contiguous Tost dite items must 


contein a large number of push- ~fetch pairs, i. rte a * page Fecently pushed 


out of core-drum is one of the most likely to ‘be fetched back ‘in soon, & 


more careful mathematical analysis shows this to be false. Based upon 


parameters derived from ‘the data accurately ‘Tecorded in "dtm 23", ‘100 page 


fetches within a string of contiguous lost data items dail statistically 


include only 4 fetches of. pages pushed within the lost dete “Stems. 


We must thus assume the worst case, that every ost data item was in 


fact a push or a fetch not properly aatched within the lost data. Hence, 


for the 435 lost trace items in "atm 23", ‘the effect “of. osing ‘this data 


could not have been worse ‘than the pushing of ‘435 never-réferenced-again 


ae 


panes on the top of ‘the extension stack, or ‘the excicias of 435 random 


potiits from the stack. In the first case, ‘the resuit 1 ‘ts that later 
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fetches are accounted to lower-numbered positions than they should have: 


been, and in the second case, some later fetches are accounted to positions 


. of higher number than they should have been. In either case, the total ef- 


fect is that of an uncertainty of either +435..or -435..0n the stack posi- 
tion axis of any derived graph. In reality, the lest data must contain an 
almost equal number of pushes and fetches. (By conservation of pages, the 
difference must be exactly the discéinaa beispen pages created and des- 
troyed in core-drum.) As a result, a typical mieplaced page in the ex-_ 


tension stack will suffer an average displacement of. 435/2, due, to the 


lost fetches. Hence, the total uncertainty. in the stack position axis of 


any derived graph is. not greater than plus or minus ong-half the number of 
lost data items. . For "dtm 23", this is 218 positions... Most of our graphs 
are plotted to a resolution of 500 stack postions.. {compared to the 
8000 or so positions of interest, this inaccuracy. is not very significant. 
Summarizing, the effect of lost. trace data is.seen both as lost accu- 
racy in counting, and uncertainty in the stack-position, axis of derived - 


graphs. Both uncertainties are proportional to the amount of lost data. 


“his typeof tnaccuracy results frém:the vpecial hamlHing of the sys- 
tem's top level diréctory pages, three th “all. “thes? pages dre ousted from 
the core-drum combination ‘presaturely,: as dictated’ ‘by a system reMability 
policy which attempted ‘to insure the integrity ‘of thest pages by keeping 
them off of the drum. As a resuit, they Were ‘Yn fact usdd“more ‘often than 
the pages legitimately at the top bevels ‘of the extetision ‘stack, snd thus, 
had no right to be in this stack at ‘all. That te to say, they would have 
been on the drum at’ almost all times had’ they ‘not bedn so bpecial-cased.: 

The anomalous effect of these Pages tad’ beak necti éazly tn the work | 
of this thesis. An experiment designed to’ dfscover' the styl ficence’ of 
this effect revealed that on some days, between‘ dhe-Half and one-fifth of 
all Multics disk traffic was a réaiilt of thesd ‘spetiat'cased pages. 

Thus, our experiment was modified to note’ ‘htt tYace data when such © 
pages were being fetched or ousted: from cove-drun. ‘The iechianion ‘by which 
this information was transstfitted ‘was ‘womeedvat! #2 Hoc; and: isnot shown © 
in Appendtx “A. Bees : wa Pea ep ee Be 2etdntetY, Bie Sr Ine! 

The predominant inaccuracy caused by these pages is a distortion of 
the very low end of the £(x) and r(x) curves (see section 1.3). ‘The stack- 
reordering inaccuracy created by abe pages cannot be more than plus or 
minus three positions (as there are only three of these pages) at any 
point in time or stack, and is thus totally insignificant. As these 
pages rightfully belong on the drum, they are usually fetched very soon — 
after they are ousted, and thus, never migrate very far down the stack. 

Thus, many cetarence counts at low-mmbered stack positions are at- 


tributable to these pages. If the core-drum combination were extended by 
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any finite amount, and managed as currently done (i.e., at the time of the 
experiment), the anomalous references would still appear outside of core- 
drum, One should thus consider these references to be to stack position | 
"infinity", meaning that they would be disk Bey Seen no matter how far 
core-drum were extended, or simply "highly anomalous", and not considering 
them at all. The latter course, which we have chosen here, is equivalent 
to ignoring the effect of these pages on the extension stack ordering, and 
considering them to be outside the domain of the extension stack, that is, 
in core-drum. The effect of removing references to such pages on MHBPF (n) 
is easily calculated. Starting from equation 3.4, we multiply both sides 


by MHBPF(n), and obtain 


r(t) 
MHBPF (EZ) = MHBPF (n)—!=—2tL—__ 


eei® () 


tomsit me 


t 
If E is greater than the deepest position in the extension stack to which 
any of the anomalous pages ever migrates, the only place in this equation 
where anomalous pages are counted is MHBPF(n). The effect of removing the 
anomalous fetches from this quantity is simply to scale it proportionately 
to the number of page fetches to be not considered. That is, if T page 
fetches (other than the "startup transient" fetches of section 2.2) were 


observed, A of them to the anomalous pages, 


MHBPF (0) .aiusted s MHBPF (E) 4 justed T-A 
MHBPF (n) 44 MHBPF (E) 34 T 


This ratio was observed to be between .92 and .96 for the measure- 


ments "dtm 23" and "dtm 21" displayed here. 
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Summarizing, the anomalous drum-abhorring pages create an inaccuracy 
of about 4 to 8 percent in the headways and headway ratios calculated from 


the measured data. 
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3.4.3  Inaccuracies Resulting from List Deletions 

These inaccuracies result from a design decision to implenent a simple, 
fast list-maintenance algorithm, as correct treatment "of these deletions 
would beauties: a eetety time-consuming fachniade: Heaca. we cocead to ana- 
lyze the extent of the mmaccurectes resulting fron this ‘inaccurate treat- 
ment of deletions. oe | 

Recall from Chapter 2.2 that the deletion Se pesee da core-drum does 
not affect the ordering of the ‘extension ‘stack. Such: a deletion implies 
that a fetch into core-drum will occur with no corresponding ousting. re 
‘this is in fact what happens, there is no inaccuracy ‘involved with core- 
drum (or "out of list") deletions. . 

The deletion of a page from the extension stack. creates a “moving 
anomaly" » as discussed in section 2.2. ALL Gaferences: to pages in posi- 
tions in front of the anomaly (which occupies the position of the deleted 
page in the extension ecack) are ‘tallied correctiy.. ‘The ‘first reference 
to a page behind ne uabaely is tallied incorrectly, because « we chose not 
to record the anomaly. It is réededd as being “one “peae cibees to the top ~ 
of the extension stack than it should tixee: heen. “However, the anomaly * ‘now 
moves down to the position of side fetch. The ‘situation ‘ts now the | ‘game 
as had the page just referenced been deleted. References in front of that 
position are tallied correctly, and exactly one reference behind it is 
tallied incorrectly, ‘and the snocidly moves ‘down. an 7 | 

We proceed to analyse the motion of auch an anomaly dows the extension 


aT Siem, 


stack. In the worst case, the page deleted \ was. at "ae very top ‘of the ex- 


i 


tension stack, ‘and thus, tile next ‘reference is "puaxeateed to be tallied 


jadetmectiy: Probabilistically, this ‘reference “itt be t to “that” ‘extenaion 
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stack position which is the mean of the eee perre f(x), mhe eeaentet 
reference frequency tate shucion: There ie. a a probability of a “that ‘this 
reference hain be as far fied the LRU a that a fraction q of the 


weight of the distribution f(x) is below tee, Now for the measurement “atm 


é 


23", there were about 2000 in-list deletions Pet 50, 000 inclist reads. 

This means that ties was one deletion per 25 dates In order to cealeu- 
late the probability that a deletion aaanetede dp past a certain depth in 

the extension stack by 25 reads, we cones ee the experinent, tried 25 eines; 
of encountering . a Tead at least that far down the stack. The probsbility 
of. success of one read being in | that portion oF the stack vhere a fraction 
q of the weight of F(x) is left is exactly ae The probability ofa 
faiiure is Q-q). The probability of - failures ts Gea) re The pro- 


bability of at least one success i 3 tries ie one mime the probability 
ae aa ae oe ied hots 
of exactly 25 failures, | or a-a-q) ys _For = = “1, te ees “the ninety per- 


centile point of £@), it is 293. For a = 05 it is Ee “Hence, by the 


time the next aetetion is | recorded, ae - quite likely thet the anomaly 
one a aa £3 ef Posie Byhs ie 
generated by the Previous deletion ie quite far down the extension stack. 
Sh adhe Ee ge Me agary at * aS eae sud Set Rg TE kien 


Hence, for the upper pereios, of the extension stack, the effect of dele- 


c 


tions do not cumlate. Hence, each deletion penerated . an inaccuracy of 


ee eed 236 sete ~ 


one stack position for each read behind it, but the corresponding anomaly 


3 ae 


moves sufficiently rapidly com the extension stack that the effect of 


[saa cago Be: 


later deletions are independent . thus, for the upper portion of the 

or te gig bag .vidas ee 
stack, the result of these Geletions: ts a a total uncertainty: of | one | stack 
position, a negligible amount, 


The above reasoning correctly inplies t that the anomalies resulting 


from deletions accumylate at the lower, reaches of the ‘extension stack, 


Pigs ae 
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and that fetches from these regions have a large cumulative error. In 
order to analyze this effect, we construct a queueing theoretical model of 
list. deletions. A deletion anomaly causes the first fetch to a position 
behind it to be off by one position. Several deletion anomalies in the 
extension stack cause the first fetch behind them to be off by that many 
positions (the number of deletion anomalies). ‘However, once this first 
fetch has occured, the next fetch from this position will be off by one 
less position, and so on, until all of the deletion anomalies have moved 
behind the position in question, and fetches from this position are tallied 
correctly. Thus, we may construct the following interpretation: The dele- 
tion anomalies in front of position p form @ queue. “Each fetch behind 
position p "services" one request. i.e., removes oné item from the queue. 
The rate of arrivals to this queue, in the worst case (all deletions from 
the very top of the extension stack) is the rate of deletions. The rate 
of service is the rate of references to stack positions behind position p. 
The length of the queue is the number of wetetand tng” ancealies in front of 
position p, which is the total error in ‘stack position by which fetches 
from position p will be tallied. Assuming seocnentialiy distributed arri- 
vals and services, with eiupeceave means ) and Li» the average 

queue length at position p is known to be L = 1/ (1-A/2) from queueing 
theory. A/p, the ratio of arrival rate to service rate, is the total 
number of deletions (assuming the worst case) divided by the mmber of ‘te- 
ferences past position p, both immediately obtainable from the measured 
data. As there were 2000 total deletions*, Gdene length approasties infin- 
ity at the point in the extension stack where 2000 references were counted 
below that point. This point is at 3650 pages depth. At 500 positions 


*In measurement "dtm 23". 
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before this, queue length is down to 4. At only 100 positions. before it, 
queue length is 20. Hence, up to 3000 pages, the effect, is. negligible.. 
At positions below that point where there .ara 2000: page fetches recorded. 
below, the queue grows faster than it is serviced. At the time the experi- 
ment ig stopped, the mmber of deletion anomalies, remaining jn queue in 
front. of position p', where p' is beyond the 2000 reference point in the 
extension stack is the total number of deletions (in the worst case) minus 
the number of references beyond position p'...As both of these quantities 
presumably grow at a constant rate, the average error. in atack position of 
a recorded reference to position p' is one-half thig queue length. This ; 
allows us to reconstruct. an approximation of the coxxeet r(x) and f(x), 
and then.all of the resulting curves, by recreating a better, more. accu-_. 
rate r(x), r' (x) as | 


_r' (x) = r(x) for x < 2000 pages... 


r'(x+ 2000-t@))_ r(x) for 2 2000 pages 


This implies that at the very. tail of the Atetrtbution, there is a stack 
position inaSeucecy. of 2000/2 = 1000 positions. At a stack depth of 5000 
positions, there is an inaccuracy of 500 positions. This does not seri- 
ously affect the shape of the sehame tin ratio and MERE curves in the 
region of interest as one can see from figure 3.6. We have re-plotted 


here figure 3.1 and corrected as above. 
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3.4.4 Other Inaccuracies 


Another possible source of inaccuracy was the forced oustings of 
Pages to disk directly from core, not. due tp ‘global transparency to the 


paging device. A close inspection of "try_‘ ‘to _write Page” in Appendix A 


reveals that. pages which should be ousted ‘from core to the paging device 


‘3 (the drum) are occasionally ousted. 0 disk, because cere: is no room on 


the paging device. This ae SAOn: avoids | Fecurston An the process of. finding 


- @ free core frame, as the lagrer process would othervise possibly involve 


ousting pages from Scum. which could require finiive a free core frame. 


Although we do not have data on the sreqeency of this occurrence gn the 
days of ve experiment, we have observed Multics at other times, md the 
: percentage of disk writes caused by auch odstinas is lees than a tenth of 


= a percent’ of all disk writes. It is true of Multics that the ratio of 


reads to writes remains fairly constant. Each read corresponds to one 


_ page fetch, and each fetch must be accompanied by an ousting at some 


time./ Hence, 'forced' oustings must be a similarly small percentage of 


all oustings, and not a significant effect. 
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3.5 Correlation between dtm 21 and dtm 23 

Observing figure 3.2, the correlation between the two plots is fairly 
remarkable. Within any reasonable accuracy for what is meant to be used 
in engineering approximations, these curves represent a measurement of 


the same quantity. 


, Our mean jheadway curve (figures 3.1, 3.3): show. distinct differences 
from the curves given in 51 for .the, mean hrendway. unetion of the Muitics 
system. We can attempt.to xationalize these differences. by. understanding. ; 
the nature of the user load during: the two different. experiments. tothe 

The measured mean headway between disk page faults, in terms of vir- 
tual memory references per disk fault, was between two and three times the 
figure see cured in Sl.. What is more, the slope of the two curves differs, 
ours starting out at almost six times the slope of the curve Sl. We attri- 
bute this to differing values of ) in equation 3.13, in terms of.the model 
proposed in this experiment. More specifically, the 'tightness' of 
working sets was greater for our experiment, aad the number of distinct 
users was fewer, causing even greater tightness of the system's "combined 
working set" at arly time. The aantiveubats given in $1 a te lmate during 
a day of very kaos system usage, in August 1972. User load at this time 
consisted primarily of systems programmers engaged in program development, 
an activity which references vast extents of libraries, tools, and spe- 
cialized procedure and data. These users were also operating without 
economic restriction, and thus had little incentive to minimize the re- 
sources used by their activities. Our experiments were conducted at a 
time when some of the Multics user load had shifted to the Honeywell 6180 
Multics system, in a state of development at that time. All of the sys- 
tems programmers had moved to the new machine ae the time of our experi- 
ments, and the remaining user load was quite light, consisting of the 
M.I.T. academic community. The lightness of user load also implies a 


smaller number of distinct users. 
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The mean headway curves which can be extrapolated from our measure- 
ments may be viewed as a function of user load / system working set 
tightness, giving rise to the family of graphs of figure 3.7. From this, 
it can be seen that a linear region on a curve eorrespondian to a large 
system working set can correspond to a non-linear region on a graph cor- 
responding to a smaller system working set, and the latter will rise 
faster than the former. If one draws the line C C' corresponding to the a 
core-drum to disk boundary, both the differences on measured headway and 
slope differences can be more readily understood. 

Another factor which gives rise to the family of graphs in figure 3.7 


is the transient response of the experiment. As the length of the LRU ex- 


' tension stack grows, so does the observed value of }. Especially when 


user load is light (smaller number of disk references per hour), it takes 
longer to develop curves of low X than high X, and this was the case in 
our measurements. Hence, it is possible that. a more extensive measurement 


could have allowed a curve of higher apparent 4 to result. 


Mean 
Headway 
Between 
Page 

Faults 


Figure 3.7 
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| Memory Size/ LRU Stack Depth 


Family of Headway Curves of Differing 
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Chapter 4 


Conclusions and Suggestions for Future Research 


4.1 Conclusion 


The most general and useful eonatuaton that can be drawn from this 
thesis is that increased primary memory aise vAeevedaei oace: fault overhand 
in virtual memory systems very sharply as it grows, ‘the ‘decrease being 
much greater than proportional to the increase in-memory. size« 

We hypothesize that the reference patterns observed, and the headway 
functions derived are characteristic of a large«ecale computer utility 
being used by an academic commmity through interactivd conseles; . The data 
being referenced on Multics: was accessed through ‘a ‘virtual memory mechanism: 
were it accessed via explicit disk requests ‘on some other type of computer 
utility, we expect to see the same patterns and headway functions. 

The most specific and concrete result which wechave arrived at isa 
measurement of the mean iawauey funétion “for -Maltics, shewing how page 


fault overhead decreases as primary memory size. approaches 4 x 108 bits. 
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4.2 The Paging Model Suggested 
The mean neatuay function BROT es: where x is oe memory size in 


pages, may be exoccased as a ‘polynomial ie3 a 
MHBPF (x) = li a jee woctece : : ql) 


Saltzer' sg measurements | suggest that | 

MERPE (x) - ay eile SE, ooh: seen ee 
is an adequate characterization of the paging behavior of Multics in the. 
range Oxx<1.3 x 10°.bite. Our experiment shows, for, the particu. 
surements we wade, that the quadratic. tem. ia: QQ) ‘becomes. significant. at 
' % = .epproximately. 2.Qx10° bits. . The trends. indicated,-by. figures 3.1 and 
ther. This would-be consistent vith the observatisa made abount arbi- 


trary increase.in primary memory size on. a.virtual, memory, system made 
above. Belady and: Kuehner (B3) -apeert MAREE (x) “ke approximate — axe for 
‘real life programs’. . In ‘measurements made,.cn TEM Wi4/44X and system 
360/67 machines, they .fownd k to take values. 'in the vicinity .of yes 
‘This model also can be described by the general representation of equa- 
tion (1) above. These observations also suggest, as does figure 3.2, 


that 
Ox 
MHBPF (x) = (a,-B) + fe (3) 


is in some cases a valuable approximation. The constant term becomes in- 


significant for sufficiently large x, and we may write 
G) x 


MHBPF (x) = Be © | (4) 
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a very simple model which is very appealing. As was shown in Chapter 3, 


this model corresponds to a ‘reference probability distribution 


pix) = e*/A | 7 | G)- 
where x is LRU stack depth and p(x) is the probability of reference ~ 
that position. This simple model of program behavior is particularly ap- 
pealing, as it characterizes "program size" as a distribution. Denning | 
(D1) has given the concept of ‘working set' as a measure of peoucan size, 
within a given time interval. Equation G) is or a more specific class 


of program characterizations, expressing the "size' of the program as a 


distribution. The parameter i may be viewed as a ‘radius of locality’ of 


the programs running, expressing in some sense their tightness’ or 'to- 


getherness'. In this sense, X is akin to the concept of working set. 
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4.3 nanswere ed Questions and Future Directions . 

The most oeviows extension: of the work presented } nee is to extend 
upward the range of ‘macy memory aise: ik ick the - nature of MEBPF (x) ; 
is known. Although the Keckaiques used in this thesis ere completely ex- 
tensible in this regaxd, “Lt is not clear whether” ‘or ‘not there: is any ‘value 
in such research beyond some point. If a certain eavunt ‘of primary memory 
reduces secondary. memory accesses to once an » hour; “for. instance, ‘the iesue 

of secondary memory access time veréua cost quickly takes precedence over 
primary memory cost versus secondary memory reference overhead. - For ‘ine. 


GL IE 


stance, our measurements Predict that ‘another seve : a mfitton ‘lords ‘of core- 
drum would reduce disk Séfecensen to once “every ‘two eretaes At this ‘rate, 
the paouoake viability . ‘of a fast disk is a greater: “{saue than the per- a 

formance improvement resulting from 1 more re corecdrum. ‘Por rateeca,, a large 


co? 


bit) slow (1 sec access time) store might be quite acceptable as a 
backing store. | ; | 

Another area of research is to fully understand the program behavior 
patterns which are responsible for models of program behavior. such as 
Saltzer's linear model and the model proposed above. We understand the 
working set model because we know that program loops, subroutines, etc., 
cause repeated reference to certain data items, and this behavior is some- 
what extensible to larger views of programs. We do not know what "causes" 
the linear model, or other such models in this sense. We can undexstand 
"distribution" type models by the same considerations of ‘spatial lo- 
cality’ and ‘temporal locality’ (M2) on which the working set model and 
the LRU replacement algorithm are based, but we have no insight into the 


basis for any particular distribution in program behavior. 
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A very major unknown limitation on the work of this thesis is the ap- 
plicability of its result.. No attempt has been made to determine pre- 
cisely what aspects of Multics user behavior were responsible for the ef- 
fects noticed. It is not even clear how apecific the results of this the- 
sis are to the LRU algorithn. 

An interesting direction of research would be to perform a similar ex- 
periment on some LRU-managed system which does not utilize virtual memory. 
A large data management system utilizing an LRU-managed buffer pool might 
be such a system. The common aspects of user behavior accessing a large 
on-line data base might show through here, as it is the referencing pat- 
terns, not referencing methods which are interesting. 

Systems such as IBM's OS/VS2 and VM/370, involving. paged virtual 
memories and multiple address spaces provide a fruitful ground for compari- 
son. In these systems, features of sharing and data addressing are quite 
different than Multics, but the amount of data to be addressed and the pri- 
mary memory usage strategy are not altogether dissimilar. 

An important direction to be pursued is that of repeating this ex- 
periment on Multics, reliably, many times, and determine day-to-day and 
hour-to-hour variations in the behavior of MHBPF(n). The thrust of this 
thesis was to develop and apply the techniques stated herein, and others 
must use these tools to correlate MHBPF (n) to whatever factors appear 
as influential. 

An interesting issue is to relate the parameters ) and a, in all of 
these models of program behavior to other observable parameters of pro- 
gram behavior and system configuration, This would constitute a long step 


toward theoretical understanding of the behavior which underlies these 
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models. 


Sekino (S2) shows the significance of MHBPF(x) in system performance 


calculations, particularly throughput and response time. Although our re- 


sults may be used in these calculations, we have not pursued this course 


here. 
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Appendix A 


ay Soe 


A Structured Program Description of Multice Page Control 


UES ER 20 oot 


This appendix describes the functioning of the fault and interrupt 
driven mechanisms within the Multics virtual memory management algorithm as 
it existed in May, 1973, at the time of the experiment. Only the paths 
within the so-called 'Page Control’ subsystem relevant to this thesis have 
been shown. This excludes some fairly complex mechanisms relating to error 
handling and the allocation of page tables. Within the paths shown here, 
however, this results in only a few small omissions. 

The aim of this appendix is to familiarize the reader with the inter- 
nal operation of page control to whatever depth is necessary for compre- 
hension of the rest of this thesis, particularly Chapter 2 and Appendix B. 
To this end, we have provided a description on several levels. 

The most detailed description of page control given here is an approxi- 
mately "structured" program, in which we have functionally modularized 
page control into 14 small routines. We have taken the liberty of creating 
a new language in which to write this program, which we explain within. 

We feel that this language conveys the general class of manipulation des- 
cribed herein with a maximum of clarity and succinctness. ; 

We have liberally etidead ob jpeee: substituting names which we feel 
are more mnemonic than the actual names used in Multics. We have also 
made minor modifications to control flow, and subroutinized routines which 
were not originally subroutines where we felt that clarity would be 
aided. In any case, the algorithm as given is essentially identical to 


the actual assembler-code algorithm at the time of the experiment, with 
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respect to state, sequencing, and side effects. 
The plus sign (+) in the left-hand margin denotes references to routines 


explained in detail within. 
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A Brief Overview: 


Multics manages both core and drum (the fate known as the “paging 
device", or "pd") by approximations to the least-recently-used algorithm. 
Two lists, the core used list and the paging device used list are main- 
tained for this purpose, the top of each list designating the Leet recently 
used page (which is the best enotce for replacement), and the bottom of 
each list designating the most recently used page (which is the worst 
choice for replacement ) on the respective devices. How these lists are 
maintained can best be ieee by reading the program that we have pro-. 
vided. The core used list containe logical Ceertererors of core frames, ir 
cluding pointers to descriptions of logical pages ss bid paging device re- 
cords when such cart tree may be asscctared with the core frame. Similarly, 
the paging device used list contains logical descciptions of paging device 
records, including pointers to descriptions of Logical ps pages and core 
frames, when such entities may be Beeoe ates with the paging device record. 

Multics tries to maintain copies of the most recently used P pages 
(where P is the size of the paging device, in vebovda) of the storage 
system on the paging device. The most Eecenely used Cc Pages eeere C is 
the size of core memory in page eee, are to be in core, as well. (It 
is assumed that C is less than P.) 

Thus, pages being ousted from core may be written to the paging de- 
vice, even if a good copy exists on disk. This fact should be kept 
strongly in mind when reading "try_to _weite_page”. iyedbe for the case 
where the paging device has no ‘COPYs pages wntce were identical to pages 


in secondary storage are never written out. “Pages | of zeros are never 


aR eh SE cola ee a ee 


written out, but their logical description ia-s9 modified that they are 
created in core when faulted. on. 

“ithe processor hardware maintains seace information about a logical 
page ina hardware descriptor. Specifically, the occurence of maege and/or 
medi ficatesa is noted in the descriptor. = 

A page fault is ‘resolved by finding a ‘page of core into which to bring 
the page, and bringing it in. Finding a page of core consists of reor- 
ganizing the core weed list to reflect ‘the latest ‘usage information, and” 
finding the least recently used. page treme, and using. “it. “Pages “Which 
have pecs marked as modified cannot be claimed in ‘this way, but are ‘writ- 
ten out. When the writing is complete, & at. some future ‘time, iis: nage will 
be in the same state as a page which has not ¢ been recently dwad or modi- > 
fied, <a will be claimed in the handling of some future. page fault. ‘Note 
that this ‘writing’ consists of initiating the avaical. operation, but not 
wetting for. it to ‘complete. It is at this e writing time that aeconlacy: sto- 
age is allocated, and pages containing zeros “are noted. It ia at the time 
that zero pages are aoeed and that aacoudaty. erorane is deallocated. 

At the beginning of page fault handling, hoigeVaeniac is performed 
on the ‘paging device, which consists. of trying to insure that at least | 
ten oteeeede are either free or in the t Process of being ‘freed. This is 
done by reer as many of the least tecentiy used ‘pages on ‘the paging 
device as necessary. “When a page is so moved, it is ‘checked’ ‘(via soft- 
ware-maintained switches) to see if it ia identical to a copy on disk. 
lf 80, it may simply be Heat incated: “from the ‘paging device. ‘If not, a 
sequence noun: as a read-write gequence (ews) mist be performed. This se- 


quence consists of dildcating a page of core ‘to ‘be ‘used as a buffer, 
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reading the page into it from the paging device, writing it to disk, and 
deallocating the paging device copy. The core buffer is then freed. 

A page fault which occurs on a page for which a read-write sequence 
is in progress causes an event known as an rws abort to occur. The freeing 
of the buffer page and the paging device page are inhibited, and the buf- 


fer page is used as the core copy of the page, and the fault is resolved. 


FO gE RES ES PRS Ae NEE ES ISIE: eS? 


The language which we have used to describe ‘Page’ Control is a bas- 
tardization of PL/I, with new primitives for some basic éperations (en- 
queue, masked procedures, etc.) and an Algol 68-1ike formalism for repre~ 
senting relationships among structured entities © = = | 

Underlined words are language keywords. Lower-case identifiers re- 


present names of subroutines, functions, or labels. Identifiers beginning 


with an upper-case character represent references to cells, which will be 
described below. Statement syntax is essentially the same as PL/I, but 
"=" is used for assignment, and "=" is used to test equality. There is 
no lexical nesting of procedure or begin blocks. 

A program consists of begin blocks, entered from the outside world 
in some unspecified way, procedures and functions, and declarations. 


declare (dcl) declarations may appear anywhere, including outside of blocks, 


and are global in scope. They define the class and type of variables, 


and the types of Objects used by the program. local declarations appear 
within blocks, and define a local scope of variables, identical to that. 
produced when a variable is used as a formal parameter in a procedure or 
function. 

The point of this language is to associate celis with values. The 


domain of values is the space of Objects. Objects are unique. Two cells 


have equal values if and only if their values are the same Object. 
There are three classes of Objects: primitive Objects, structured 


Objects, and set Objects. Within each class, there are different types 
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of Objects. Objects have no names. Only primitive. Objecta can be referred 
to explicitly, i.e., other than by reference to 4 cell. having the. desired 
Object as a value, or a function returning the. -desired Object. . 

Primitive Objects can be of three types. ‘The firet is. boolean. 
There are exactly two boolean Objects s One can be. referred to explicitly 
as true, the other false. The second is grithmetic. There is.a first- 
order infinity of these objects, which are actually the integers. They 
can be referrred to explicitly as. 75 16779216, 7283, ete,» The third is 
literal. They are simply arbitrary: primitive: Objecta,: whose only useful 
property is their uniqueness.. They can. be referred: to explicitly: as "foo", 
"bar", "no stuff", etc. They are. pot. character: strings. in any sense, but 
simply unique primitive. Objecte of type literal. 

Structured Objects consist of a finite member. ef cells. Any.cell 
can have as a value only one typeof. Object: (implied. is one class, as. 
well). These cells are called component: of the Object. . These. celis.do | 
have names, and they are. specified -in w declaration which deacribes. the 
concerned type of structused Object. 2 26)) 0 85 a.6 

Set: Objects consist:-of an ordered, set of Objects: of: the same type 
and class. All references: exnept: coguse. and deguenan however: consider. 
the set Object as unordered. One can .@éd: to or. gpyyeue.to a set Object, ._. 
Temove from or dequeue from it, ask if a given Object is. a member of it, 


or cause a cell to be assigned succesaive:valsen, each. value being a dif-.- 


ferent Object: in the sat Object, .in no 
'Naxiables are the other type::ef. cell... A.warieble.can bold only one. -. 
class and type of Object, just like the other. type of celi,. the struc- 


tured Object component. 
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- Assignment (performed by ":=" operation in do: statements wad assign~. 
ment statements) consists of replacing the value of a:cell with another . 
value, i.e., changing the vaiue of the cell. The Object which wes the pre-"" 
vious value is neither changed nor destroyed in any way. °° 

Binding consiste of saving the value of -d variebke when a procedure, 


function,.or begin block is entered, and: restoring it:when it is exited. 


The latter operation is called unbinding. All: assignments and bindings . 
made between the time a variable is bound and: the’ coreesponding unbind ing 


have a transparent. effect: when the block: performing the: binding is exited. ies 
A local declaration of a variable in a block’ causes:‘such a binding to take ~ | 
place for that variable when the block is-entered;: and the corresponding 
unbinding. Binding also takes place for yaviabtes ged as: ‘formal parameters 
to procedureg- and functions. In this case, after the old value is: saved: 
the value of the. corresponding formal argument. is: assigned to. the variable. 
Hence, all calls may be seen sé "call by value". FR 

To refer to an Object, one can‘either refer:te. 2/sie bb containing ity. 
or, if it is primitive, one can refer to it -explicitty.: To refer to a 
variable, simple state its name. To refer'to a component. of a structured 
Object, state its component name, ai open! parenchestsyir reference to the 
structured Object, and a close parenthesis. — 

An assignment is a reference to a cell, ";=", and @ reference. to an 
Object’ of the same type and class declared for that aeil.: 

Variables need not be declared. ‘The default clags of any ceil is 
structured, with a type the same as its name. The ‘syntak for a structured 


Object type declaration is ag follows: 


91 


(eset) structured Foo (compdel-1, compdcl-2,...compdel-n); 
[ ] = optional { } = select one 


The compdcls, or component declarations, are of the same syntax as variable 
declarations, except that the name is the name of the component, and the 
optional keyword variable is illegal. 

The syntax for a variable or structured Opject component declaration 


is as follows: 


fae rsa [variable] Foo [type] {objtyp] 


where objtyp is either boolean, literal, arithmetic, any structured Object 
type named in a structured Object ‘type declaration, or set objtyp, where 
objtyp is, recursively enough, any possibility named in this sentence. 

local declarations only name their variable, although they can declare 
its type as well. 

do statements differ from PL/I in that any cell can be used to the 
left of the ":=", not necessarily variables. The particular form "do 
Foo: = range Bar" means that the value of Bar is a ‘set object, and the 
do is to iterate over each Object therein, in no special order. 

The special constructor function construct is used to create new 
structured objects. The syntax of a reference to it is 

construct Foo (compname-1:object-1,compname-2:object-2...), 
whose value is the new Object. 

The unique Object "null" can be used as a value of any cell. It has 
all types and classes. 

The predicate void takes as an argument a reference to a set Object, 


and returns true or false (boolean Objects), depending on whether or not 
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it is empty. The operators "=" and Lal may be used to test if, two refer- 


ae 


ences are equal, i.e., eeeeeS to me same Piece. An appropriate boolean 


Object is returned as a value. The operators "OE" s "and" and ' pot" 


pe gest. 


operate on boolean Siekts in the obvious way: The ‘conventional arithmetic 


spevatete operate upon arithmetic Objects, returning 4 an 2 arithmetic a es 


with the Sarectes: value. 
if statements have as their redicaks a <patevenie to picleas Op Jecks: 
A call statement consists of the word call followed by pee a pro- 


x: function: Teference 


cedure name and an optional argument ligt or: a co : 
and an argument. list. An argument list. is a. _Parenthesized list of (pos- 
sibly zero) references to. Objects separated. by ees: A complex function 
reference is. a function reference. to some, outside-of-the- language function 
which will return as a value a procedure, which gne depends on the argu- 
ments to the function, which will be called by the call statement, with 
the. arguments to the call. | 

‘The evaluation pf arguments in or ang and As. gonditional, as in 
Lisp. 1.5 (3) and proceeds from left to right. : 


and Have Him and His Father Switch Houses 


declare structured Person 
(Father type Person, 
House); 
declare structured House 
(Color literal, 
Owner type Person); 
declare Son Person, House2 House; 
declare Brooklyn set House; 
switch_houses :begin; 


do House := range Brooklyn; 


if Color (louse), = "black" then da; 
House2 := House (Father (Owner ee 


Son := Owner (House) ; 
House (Son) := House2; 


Gwner (House2}-:= Son; 


_ House (Father (Son) ) t= House; 


Owner (House := Father (Son); 


return; 


/*assumed to be initialized*/ : 


[ {search the set "Brook- 
1? lyn” 


A fgged him 


//find the other house 
aiken who is the 


v/s son” 
_ £/Sen new: owns ; House? 


//Father owns house 


SST SEA erry, 


A page fault causes the following 7 | (page_fault) 
The paging device is housekept. ee oe = 
Transient conditions such as i/o in progress or an rws on 
the faulted page are noticed and handled. . 

A free page is claimed, and the faulted page is read or created 
into it. 

If i/o was started, the page is waited for. 


i ta es ee mE aa 


Finding a free page consists of the following: (find core} 

The core used list is searched for a good. candidate. . a ee 

Recently used pages are not good candidates. Rosser re skipped, and ; - 
re- judged as not-so-recent ly used: for next time. age 

Pages which have been modified (stored into) cannot be claimed now. 
They are written out, and re-judged as not to have. been modified. 

A page which has not been modified, and has been used ‘approximately 
less recently. ‘then any other page, is ‘pre-empted ‘from its core — 

frame, and this core frame is: the: sew ‘free ‘page eee. 


; (write _page) 
‘The page "s "contests\ are checked, and if aii's corsa, ‘the page is 
flagged ° ‘as not needing to be read or written --No writing: takes 
place, and disk and paging device space.alloceted to the page 
are freed. : 
The page is given a residence on disk, if it dees ‘pot already have 


Yor ide tess fhe tues 


one. 

The page is given a residence on the paging device, if it dges not 
already have one, and one is available. - 

The page is written out to its residence on the paging device, tf it 
has one, otherwise to disk. The completion of i/o is not waited 


for. 
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Housekeeping the paging deyice consists of the fol lowd ng: (get_free_pd_record) 


An attempt is made to insure that there are ten paging device re- 
‘cords free or being freed, which is done as follows: 

The pd used list is searched for a good candidate to pre-empt. 

The search is made starting at the least-recently used pd record. 

Records which contain pages in core are recently used. They are re- 
judged as such and skipped. 

Records containing pages identical to pages om disk are acceptable. 

The pages in them are pre-empted, and the record is now free. 
Other records have to be written back to disk, which is done by 


performing a read-write sequence (rws) on them. 


d-write sequence on Jc iL of the followi 


(start_rws,rws_done) 


Per formi 


A free page of core is obtained. 

The page is read into it. from the paging device. 

When the read is completed, the page is written out to the disk. 

When the write is. completed, the page of core and the paging device 
record are freed. 

A page fault on the page involved in the sequence at any point 
during it cavees the sequence to be aborted at the next complete 
operation in the sequence, and the core page is used as the 


page's home in core. 


A Page Object 


A Descriptor Object, . 


A Coreadd Object 


A PDrec Object 


A Devadd Object 


An Io-status Object 


An Io-program Object 


fs the’ logi¢al description of seme page of the 
storage system, as opposed ‘to a page frame on 
some device. mee es 


in actuality a "pagé table word", is the physical 
descriptor ‘by-wifth “a processér necesses « page. 


It contains a core addregs; tage bite, and a bit 


which causes a failt when‘ off; 


describes a physical core block. It describes 
the status.of this Klock; inéluding, implicitly, 
its position in the core used list. 


‘describes -a'paging ‘device record; ‘or frame. It 
describes the status “of  thiv frames -ineluding; im- 


plicitly, its poésitfon°in ‘the paging device used 
represents a phystea) disk or deum ‘address, and 
its contents. “Ineludéd in this‘ebjeet. 19 an iden- 
tification of the device*ée°wiich this page frame 


resides. 


is a hardware-generated object, which describes an 
input-output operation which has completed. 


is a sequence of commands for the system i/o con- 
troller to give to an i/o device. It specifies 
the type of operation required, the record within 
the device concerned, and a core address con- 
cerned. 
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A Trace-Datum Object is a recorded datum of information about traffic 
between disk and core-drum, for the purpose of 


the thesis experiment. 


je Description of the structured Object types used by Page Control. 


Recall that the default type of a structured Object component Is 


the sama as. its name. */ 


gol atcuctured Paxe 


(Descriptor, 
Devadd, 


Coreadd, 


PDre¢, 
Event literal, 


lo_in.progress boplean, 


On_pd 
Wired 
Gtpd hp 3 


dol atcuctured Descriptor 
(Phys_Coreadd arithmetic, 
Addressable boolean, 
Usage hoolean, — 
Modified heplean); 


Sol atrcuctured Coreadd 
(Page, 


Phys_Coreadd arishostic. 
Next type Corea 

Previous type Coreadd, 
lo_readlor_ write literal, 


Rws_in_frame boplean, 


Represents a page of sone sexment of Multics, as opposed to 
a page of core or some device, 
The hardware descriptor by which processors access the contents of Page. 
The physical disk or pd address from which Page should be 
read or written to. If On_pd ts true, is a pd address. Otherwise, 
tt Is a disk address. A Nevadd of "null", however, represents 8 
page full of zeros. 
The core frame associated with this naze. Valid only when 
Addressable (Nescriptor (Page)) Is true. 
tf Oncnd Is true, this Is the pd record used by Paze 
Some [lteral quantity unique for each page. Used to Identify the 
eccurence of events associated with this paze In Interprocess signaling. 
Truth Indicates !/o In progress, or at least not known to have - 
completed, on Page. 
Specifies that Page has an allocated PD record, namely POrec (Page). 
indicates that Page must always remain addressable 
indicates that Page is forbidden to go on the pd, for reliability reasons. 


2] 
io) 


Represents a pare table word (ptw), the physical descriptor by 
which processors access 8 page. | 
The physical core address occupled by the pase to which this Descriptor 
belongs. Valid If and only If Addressable [s true. ayes 
Truth allows Phys_t Coreadd to be used hy the processor, Falsity 
causes the procedure "page_fautt™ to be executed. 
Set by the hardware whenever this Descriptor is used, 
or more accurately, fetched Into the associative memory. 
Set by the hardware whenever a store-type cperation is 
performed using this descriptor, or an associative memory 
copy thereof. 


Represents a core nace frame. ; : 
"null" represents unallocated nage fr Otherwise, th 

Pare contain ed in thls frame. #012 fs onty tor it hed page-holding 
use, not rws's 

The physical core address represented by this frame. 

The next more recently used core frame. 

The next least recently used core frame. 

if toltIn progress(Paze(Coreadd)) Is true, or Rws_iIn_ frame, 
tells which direction of t/o Is being performed, 

Stgnifles an rws in progress tn this frame. 


PDrec); 


del structured Devadd 


acl 


(Nevice 

Phys_Deva 3 
Pores 

Diskaddr Syee Oevadd, 


Devadd, 
Coreadd, 


Nextipd type PDrec, 
Previous_ia type Pirec, 
Event Literal, 


tn use bealean, 
Rws_In_prorress boolean, 
Incore bealean, 


Modified from_disk boolean, 


Abort flag boolean, 
Abort_complete boolean); 


aa cea ee feed 


Phys_Oevadd arithmetic, 


Phys_. Coreadd 
Next” type te apatite 


Og) aicuctured Jo_status 


scl 


(Phys_Devadd ‘6 
Phys_ ~coreadd 


fo program, 
Coreadd); 


Trace_datum 
ievadd, 


Type literal); i 


//) Used only If Rws_in_frame Is true. Specifies Poree having an ews. 
4? Represents a physical device address. 

//— Identifies a secondary storace device. 

4/) identifies a phystcal record number on some device. 

4/> Represents a paring device (nd) record, 

4/,) Vf Incuse ts true, ceseribes the Pare on this record, 

// Vf tnluse Is true, describes the disk address occupied by our page. 
7/7) The physical device address of this record, 

// When Rws_in oronress Is true, deseribes the core frame 


// being used as an ews buffer. 

//) fescribes the next rore recently used pd record. 

// Mesertbes the next least recently used od record, 

// A unique lizeral associated with this pd record. Used to Identify the 
4/ Yatter in tnterorocess sirnating. 


47 Yells tf this od record Is In use or free. 


// Stagnifles that an rws or rws abort is in prorress In this ed record. 
4/) Stanifies that the paze In this od record Is In core right now. 
// Used for matntaing the LRU ordering of the pd used list. 
//) Truth Indicates that the pd copy of page is different 
// than the disk copy. 
//] Turned on to start an rws abort by some process faulting on an rws'ing 
J/ Pare. 
//° Stentfles that post_oare (q.v.) has aborted an rws, and a cleanup 
/ by rwslabort (a.v.) Ts exnected. 


//, A eertion of a channel program. 
f/ Indicates read or write. 

4/7 Physical device address Involved, 
// Physical core acdress Involved. 
// Next pronram in channel queve. 


Ji) Represents a completed t/o operation to an 

4/ to eantrol routine. 

vy) Identifies the obhysical device address involved, 
tdentifles the physical core address Involved. 


1 Bes TAs RE Hear tt in torneo, atenoun 


// not actually present here, the one-to-one mapping between 
// Coreadd's and valld Phys_Coreadd's jets us use this here for 
Wy clarity. 


4? An Vtem of trace data for the experiment, 
4/ The disk address concerned. 
‘The direction and description of togical motion. 


4 
“read” -- @ pare fault to disk or an rws abort 


4/7) “write” = an ews Initlation 


4 


“virtual™ - an ousting from core to disk 
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The Global Variables Used by Page Control 


del 

Page table lock literal, A quantity used to insure that only one pro- 
cessor at. a time is in page control. A pro- 
cess desiring to “hold" this lock loops con- 
tinuously until it. is unlocked, and. then 
locks it. 

CoreTop type Coreadd, The least recently used Coreadd Object. 


Writes_outstanding arithmetic, . 
The. number of write. ésperatioas started which 


have not yet been known to complete. Used 
as a heuristic to call post_any_io. 


Rws_active_count arithmetic, my. number of read-write sequences which have 


been initiated and not yet known to be com- 
pleted. 
Number_of_free_pd_ records arithmetic, 


The number of paging device records free or 
in the procees:(mws) of being freed. 


Top_of_pd_used_ list type Pdrec, 
The least recently used PDrec Ob ject. 


Channel_Queue type Io _ program, 
The executable queue of i/o programs for a 


disk or drum. 


Experiment_active boolean, ~ Tetls if meter ing “experiment is in progress. 


Preece queue. Sele trece satay The total of all trace data accumlated by the 


experiment. 
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Undocumented Routines Referenced in this Program 


page_wait (literal) Suspends the calling process until a call to 
notify is made with the identical literal. 25 
page_wait alwe-unlocks the pap table lock once 
‘the traffic control data bases are locked. 


notify (literal) '  Gauses any process which called page wait 
with the identfeal literal to be resumed. 


clear_associative_memory - Causes all procesgors to claéaxr: their associa- 
tive memories. This routine does not return 
until all Processong: have: Dadicated that: they. 


have done ‘3 a0. "Used to force access turnoffs 
“and. modi fied “bie: ‘turnoffs to take effect. 


allocate_disk_record() Returns an unallocated a Object. Marks 


relinquish_disk_space (Phys _| Devadd) 
Marks a Devadd Obi jeck as unallocated to allo- 


cate_disk_record. . _ 
start_io(Io_program) Starts a chamel ¢xecuting an i/o program. 
thread_to_top(Coreadd) Changes core used Iitst end value Gf CoreTop «.. 


such that ‘Coreadd ‘is moved to the top of the 
Core used list (least cheer aes 


. KoxeTop. now = eae BF sok. 


thread_to_bottom(Coreadd) anes: core used ‘Mest and value of CoreTop 
. such that Coreadd..is mayed Q the bottom of 
the core used list (most ‘recently used). 
Next (Coreadd) now = CoreTop.. . 
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Core Map PD Map Page Tables 


(Coreadd Objects) (PDrec Objects) (Page Objects) Page table words 
: (Descriptor Objects) 


J 


Core Used List 
ie Sse] 


PD Used List. we : we 


Previous_pd 


on | Descriptor 


fr Soxsedds ese 


‘elec 
Bor mryoo x 
oro ® w Pp oe} xa oP 


e . 20s Fert ¢ a 
OD 
© 

. 2 7 


In _ use 


Rws_in_ progress se —_ sae 
Incore | ; | To_in progress enue 
Modified _from_disk | TD Foangl 
i fe Wired 
Abort_flag Ie Lb 
Abort_Complete if Inot;” we, Gtpd 
a on| PD Disk Record: 


er 
Io_read_or_write Corum | 4-A 
Rws_in_frame 4 “ . 


Core Page 


The Page Control Objects for a Sing] Page ‘Figure A.1 


Or 


page fault 


Scu.data Virtual_reference; 
Virtual_reference (Page, Segment); 


page_fault: begin masked; 
- local Page; 


set_lock Page_table_lock: 

Page :* Page (Scu_data ); 

do while Number_lof_free_pd_records < 10; 
gall cet_free_pd_record; 

ends: 


LE Aé@vessable (Nescrlator(Paxe)) 
then unlock Page_table_lock; 


é 


then call page.wate (Evene(Page)); 


fons 


else if On, 
then 


4 


d (Page) and Rws_in_progress 


e 


call rwslabort (Page); 


else If lo_in.prograss (Page) ; wie 


// This procedure Is transferred to when the 

// hardware determines that a reference has been 

// made through a Nescriptor whose Addressable bit 

/t ts false. Return from this procedure causes 

// a second attempt to make that reference. 

// This procedure ts entered [n such a way 

// that all external (1.e., I/o, etc) Interruptions 

// are disabled when It Is entered. They are reenablted 
// when It is exited. This fs because such 

// Interruptions might try to lock the page-table lock. 


// Prohibit access to page control by other 

// faulting processors. 

// Determine from processor state at fault time which 
// page was faulted on, 

// Housekeep the paging device = try to have some free 
/1 pd records for the find_core calls which will 

// surely follow. 


SOT 


aT Do a re WR 

x oH ofa re: ra | ee RS OR gin SES LES AE 
// tt 18 possibie that we took a pase fault while the 
// page table.was leeked, and the process holding the 
‘(ecg Brougne Eke page tn. Exit $f eblg:ts true.*: 
- th bs. noggtbte that we took a cage fault on a page 
édcwhl ¢h<gome other pracess has stare ted@br fneing fi. 
W/ walt fort : etwid 


ARQ MG Pak Pw 


eeu es 


a ree eae OO 
7M  \The | fvgkes, routing .page_walt causes: the: 


/ .guspenslon et the calLing processcusel!} © * 
// some other meocess cabts «the: reutine ‘notify 
// with the tdentical Event with which pare_walt 
// was called. -Pagegeatt also unlocks the 
‘1. pare table-leck ance he has locked his data -bases, 


eet Ore Cok ae oe Po RR PM a MR ET 
CPhrec(Raged) =, / J “ER. th spage Ks‘on the paging 
+t dawice, \t Is possible that.'8 readwr tte 
// “sequence may be In progress for It. 
// Vt must be aborted, 
// Abort the rws, or possibly clean up an 
// already complete abort. Unless we are 
// cleaning up, we will wait for tt. 


Lf Rws_in_progress(PDrec(Page)) // \f we have cleaned up, we are 
// finished. tf we started an abort or 
// noticed one In progress, we must walt. 
hen call page_walt (Event(Pfrec(Page))); // Walt for the abort to 
// Complete. 


end 
else do; // Normal case - we must bring Page tn. 
call read page (Page); 4/ Start read-In of Page. !f pare is empty 
// (all zeros), or was on a fast device, we may be done. 


If to_tnupropress(Page) // If real i/o was started, and not finished, 
then call page_walt(Event (Pare)); // we must wait for post_page 
// to post Page. 
else unlock Page _table_lock; // Page was all zeros or on fast device. 
> o} // All done. 
end; 
return; // Restore machine state at time of fault. Retry 


// faulting reference. 
end page_fault; 


90T 


read_page 


read page: procedure(Page); oe : . J/- This procedure ts resoonstble for 


4 cay Ing a faulted page to appear In core. 
Véet /o ts necessary, it Is Inithated. If not, 
lia "pane of zeros Is created. 


dacal Coreadd;.. 


If Onpd (Page) - We Aitoun: ‘this check. is made fa: pane. faut, 
then if Rws_in,pronress. (PDrec. (Paze)) // te. 1s conceptually: important that:it be made. here, 
then sasschae ee ee are // for cqad pare may be: calted by other’ system 


// functions. !f an ews ts In progress on Page, 
//.we,.ganngt read it, :end our caller must efther 

ee // -glyecun-or inittate abort proégeedings. 

Pad //. Allocate. a: page-ofcere for the pare. 


ae (coreag) .c = Panes Zi Indtcateithac thts pane detdriad: herd: 
tl age) neni" //- A null devadd Indicates that s page is defined 


// to contatn zeros. 


eer aca rei ¢1026 words) := 00000.... 3// Make tt all zeros. 
: = er 4/-— tndlcate that this frame belongs to Page. 


ire 
reeeT 5 cae make_acce: Sstble tare); // Reset fault bit In the Sescriptor, allowing 
ee cy Leaeas ¢/ et ptt to reference Page. 
Oey, Push fb ee Ae IEG Sk PWS” 
io! (Pane) i= trun; i hat ieeae tied tie: "ty prokrere. ets 
oLinuprogress (Page) :@ ey e roxréss. ft ts - 
to_read_or_write(Coreadd) :* “read”; ean fibeg ene ee dene befor ore Ue tte is a 
oat) ae ghe <ifo peetine ice ean reset these 
fie ep tintshds earty. 
1 device read (Devadd(Paze), Phys_! op a dsStart: the ‘read. : 
npd (Page) nd managenent know poeta this. page.” 
then do; 


Ske = dacore(PDrec(Page)) 2s 3 dp has a: i tn core (racantty used 
1 pdithread_to_bottom(Pbrec(Page))3 7this Vine Is not te the Setwal code. 
. ‘hts. absence const itutes'a bug which was found 


Pe 
nee ter. it helps maintain the tRU ordering - 
a “4 of the PO used list. */ 

2nd; 


end read_page; 


LOT 


findcore 


find corer functlon(); This function Implements the Multics core 


He age replacement atgor i chm, '5 Ts’ catTed to 
. fousekeen core. and return one free Coresadd Object. 
dal CoreTop Coreadd; , vie ‘Ite basic data base Is the core used 1ist, which 
Lb iy Is the eresr ion of Coreadd Objects ot 4 by the 
1 Sore quence, of Hext. componests of Coreedd 
us variable CoreTop has.as. a value the pec 


fgetiex etter efepiie and'se tens ere 2°) oreadd 


Jeets having seen. py ip recent’ use. The core 
Adet is clreuler, so. Nex frmpt Tec becently used 


eadd) = CoreTop. The algor 
i orate. 
‘Frome type corendd; en eho et 
Pane; Pete % : ag 
Josal Igabione puter. aelinpestc 88 - 
sent hg : -_ 4 --tatelalize cheek for excessive looping. sad 
eepde geet ot Bie = ar We. search the used: dist as. long &s necessary. 
p_tou {man efault is:te tresh Multicsi:: 
(Lager eolitar < * 131072 3 We; Au & way? Pesan: s&s free: oane, orvdtny 
p pe Ve: Bis te fy. 
os Writes. Poe aa > oe . bbe tes ni rowttng 9 eunued ibn rites see If 
ae POSS pay.193 af “f§ ane apcyes au ae. seer et leepings i 
Mees res : “47 irene fet eret totecruses: arevmpeked) § 
Ve toskipcounter := 0; 4 intttalize te check, Wg have just done what 
vo cy, es . 17 this check could ask us to do; 
i : 4 # ro i ee ca 
Page (CoreTop) = “nult" : 4 be, edly: Jeessccecent}y.used page to 
uf than do; ‘43 fates ayponye f frtiise shied teow d happen by 
, fws, of & paxe Relations. tf so, we take it. 


eSre ob" S482? Leoraroonsff oR Merk eH Py Partai of ert recénety used, and the 


// next least recently used the teast. Notice that 
this common operation Is trivisl only because of the 
iy clreularity o the core used ific . 
return Frame: // ~=-Return this core frame as shastie’ 


end; | 
uf toiniprogress (Page (CoreTop)) // tf, In the next }ine, we are going to skip this 


// frame because of t[/o, other than an rws golng on, 
// meter the times that we have done so. 
ghen ltoskip_counter :* lo.skip counter ¢ 1; 


lf to inuprogress (Page (CoreTop)) // See If we must skip this frame due to I/o 
ar “Rws_. In frame (CoreTon) // In progress there. 
then Lf To ~sk 10. counter ¢100 41 If we have skipped a targe number of pages due to 


// to '/o tn them, see If any I/o has since completed. 
Gall post_any_lo; 
lo. dice counter 3:* 0; // Reset this high-water-mark. 


end: //- = Repeat the loop, trying chis last t/o-skipped 
° // page again. 
else CoreTop :=* Next (CoreTop); y eet aver this core frame, consider Next to 
e LRU. 
egise if Wired (Page(CoreTop)) // Skip over pages never claimabte. 
hen CoreTop := Next (CoreTop); 
nie if beaker rcr rt (Page (CoreTop))) // See If page has been recently used. 
then o 


ea a 2@ false; 
sc py if $0 rales tarts Peat for 
‘ineat time, and skip this wo 
{Joane, making 
// Wt the most recently used. This 
AGG Usage bit Is.set true by the 
/{ hagdware. when. the. descriptor Is used. 


ay CoreTop := Next (Corefop)s.. 42... Sklo over: i, ee = 
alae go; // At this steges.the-oege af the top of the © 
// core wae lle tohes sbgen, recently; used. : 
// itisa ef targes_ Tor ceplacement. We will P 
‘s le it, needs. ta he-weiggeavout, . af 29 | 
‘ts 


rPragress wren: § ory 86welte pase: 
// returegs. we can claim the page. 
Page :* Pagze(CoreTop); “(aust der femme? .cuErear: resident. 
Frame :© CoreTop; ig 4 And oe er. abe: freme.: 
CoreTop := Next (CoreTop); ecusder consideration 
ees Sat: Ba tly used: if.we ultimate: 
; i} lgrcoletme &,* bred was: agi right. move. 
fe VEower-de-aets it. wll £0. recent 
7 cea ae // use or t/o, and it Is ert good. 
call try_to_write_page(Page); // See If pare needs writing out. 
fi: Unitlate such tse tf so. 
If not to_In_progress (Page) // Nf. tey= “tegwriteeage succeeded In 
/{ totally wetting out» pare -¢fast pd), 
{1 this-hotds. Otherwise, move .on. 
Shen do; Ef Tey-to:ckatm oare for real. 
call make. honeccesaivic, Nebihe Fg 
// Turn off access to Paze. We do not 
// extt this call until all processors 
// have verified that:they have flushed 
// Deseriptor (Page) from 
// thelr assoclative 


ih apa te Sn 


wd: i hag 
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_ tey_to_wrel te page 


try_to_wrlte_page:procedure (Page); 


gdcl Top, of pd used_itst mcees 
Lf. On_pe. (Pare) _ 
then: Lf. todt fled (eeseriptér (Paxe)) 
else 
OH. Gtpd (Pare) 


Qe Iniuse  (Yop_otipd_used_ vet). ea: ef 


* ghén uF Moat FleatoesEr Hbtor (Pane?) 


4/ Thts procedure determines If a page has been 

// modified, and thus needs to be written out. It also 
// checks for the case where a paxe should be written’ ‘to 
// the paginx device due to recency of use. 


tt if the page atreaty | has a copy on the pacing device, 
/ the decision to nee nei tg the same as whether of not 


/ 
h 2 pau _welte_page (Page, "modified", nods ok"); (7 Page h bego ‘@odifeled. | 


— 
Page not moultted,. ready on pd, need not write. 
// Page ts not on are pagtng device. If It can go there, 


uy we wilt “} aq 
caf make ts "6 a fects den’ te fO‘on pd, 
fe ‘TS ho space for it, 


we capo a e ate modified, 


n gal 1 wrlte_peae(Poce,"eodif led”, 


end try_to_wel te page; 


jéa, ¢ e 
iy rel se if eepreeet Use. oe mitt fled. or not. . 
It Modified(Nescriptor(Page)) // 


"BAR ABH GHERBME eee ocogteltt ato 


moat t| id ‘Tadicare it and write it. 


"ys "17 Welte even If not modified. 


2 WRAY Steen ie, 


wel te_page 


write_page: procedure (Page, Modflag, POflag); 
// Thts.. procedure, which Is called when It Is 
// determined that a pane must be written out, 
// does so. it ts defined in Multtes that 
44 2 pare of zeros ts never writeen raged but spectally 
(| Hire ts Boe Me. make. that check her 


i ; iste ; pale e rest “here.” Modified 
$. git ~ : telll 

(/ whe or “opettle a on nt ag tied fre ndtsk 
it CroteetPerey) ¢ thes, ton_pd", tells 

tl us a allocate a rie od fecaee for Page. 


declare Modflag literal, POfieg Litgral, 


: ot 7 : fs ope : par it 

If pageiis_xero (Page) “then geturn; Hs a fev tenes zeros, fio write need be done, S) > 

Modified (Descriptor (Page)) = false; a] optke hy ge to Page. Any : 

my ae oe , 4¢ aie ¢ thls petnt (actually the next g 

i eal i 9 téd by f g_core once the I/o that f 

a, is t nar Fratsned, 4 

Paty cledesptegclative.pemory: | off, “tats sick 4  oedces ters wt iy itty, ‘thts pagé ‘note 4 

. tata streagvess¢nage) fa: is FRE "a eee chee (and “nee ‘by “obese t$ ‘ngt¢ étatmable. ‘ 
& ger | ce(Coreadd ore) te eer Ld * EE pd et Sage “Kron Wrat “kind Gf T7o took place. : 
- if Devade (Page) | = “nut // 1f page has no secondary storage home, 1.e., was ; 


Tosa pa nes f TE! o bys 
it 1 BH Devadd (Page) is ‘allocate wdiskrecord(); // at at! zeros, give It one. 
a 


then ae} “tists iesaay: = f z reeahOe" ate Rie recency Ae use, 
fae Page oy Ben coh ig ad Raacd th Aare Sn os ¢. (prec 


sais . 4 € 
0 EF On BS Crdbe) rt SPP Ae iF he re, at Ontd Page, or’ te ‘dd, 
shan // then update the status of Its nd record. 
Tf Modfiag = “modi fle // Af Page has heen modified while In core, 


then Medi fled from disk (Phrec(Page)) 2° feue; // we must Indicate that the pd copy that we 
/ are about to start writing is different from the 


cal) Pdthread_to bottom (PDrec(Page)); 4 thafeete that this pd record has seen recent use. 
Qe RE SE ge 
11 device_write(Devadd(Page), Phys Coreadd(Coreadd(Page))); 4/ = Stare the actual write. 
end write_page; 


(Geto desma y\ 
aOR eS: See Teo 


S aes Meth celwont ion if 
fs ad Heyer geet peng Tun? 5 (eged} bone 
oied zeegtedn! sage sess 
oh sebves tae) ae 


Caer there ier Riete ona A ouk Li aucert 


* 


Pee pate es matt bet les wre wo) DE 


“ES 


é 
vi 
Ry 
+. 


114 


2ACY 03 P2sep|SuED oq VED exeg = // " a4neg s¥ed 0y2 wosy potter esen om) JT 
*peisadue $1 So4sz 30 BdOd & // 

30ys VOTED; pu, VE Se ppRaep 4iAY // 

ShY2 SIssd4a2u; szec"pees *souar Hy 

Ugh, of (8%8g) 


Wee Y i ese p peasy) soe asi Pusinpul yoo tree 
We 


#8 Dealt svearoeyena) nS ip"s03em TTES 


LOS sg ogc 2? 


1, ey 
oar vase a See gl sate as ies 
Suywreis ere ;peum; 203 psodas p 
one : 46g )9e. S83 LY 115 i 
ase ieee 4s HS veeesiad tyes squy « wily gua] os 
apes paedes Bas, Meas os Ww Spscons "pa 8035 seamen Repti eae 
ss aiewg a i 
“sposn oq sew psenss po Bun 47 fértes”. ey eee a 
*ySip 8] Swoy, k0aegy ted SUEPEAID BGA , 17h, asAHly,°! # wee. ppeaay | 
Sede 01 sald | j1 *psoden pO Ody _spneutege' 11 codeaypeto ry 


Pe hein WA 

BALE DOSSe aur sH0(d' Bod *paow OM’ (UbZER WES PETES 
UdIQ VUSARY B2dG 4d JIeMY B4E BM 2EqI pth ae 4 

et i sha) saraianens) 0813 1PeW 


es at rete Skt! « sideman TT 


pve *esaz ae ose ip rs} payseur a 3o seat y 4 ; 
SEA BY BENEQIAG BUY pa ave, a9 wan 
AlOs1S 5O.peazsys 94542 : e asey Ml 
sey axe¢ on ele 0) 3¥ ie Si slog em yoo) 4/ 
L118, an 880d a4 so AAILIges eappe 842 ise wind i 


£(e8eg) siqiseapoceer ayes ie 


109 WT 
fe ss0z07. wie be. 
Auye21shye ave) asec $1 00g Nn “*"@ “a (Spam qzet) co cccoueg) ppeesod) ppeeses"sAug) PI 
*paysynbuyi as $s) sted eya // 
m epretosan tie dutfaes suesy case @ aoe s6 1 : 
1 a * Bod ‘. 
soars @Uywsa3ap O02 pated 8} Sunpacose stun) of $(e8eg) UOLI>UN, t0202"s “ORES. 


Os02~s | “axed 


wie gassianl a hedian ei aims sa%d ant AN 
ot eolagatlen tf. pehioges fe oe) Go sodmun AL 


bagvesa red ene Bees). gat 34 Goi este AL 
sete .geonmvee: we lte-heas Yq gadmun. gazed 4 
we thdd ansexibal ssaevace wiliwebewt a eat ssage 44 
; OT euee wow? ata ,otdallaews ed [fiw Sieger sc AS 


* . 


vabtariree 
wo gad noe) ob 4) 


sae waded ied Ye. Found oats wet ingastnt ue 

‘ eeeeenen 26 grat en adel ah regee? AA 
oy 76 eal ve “qdelt bees bo at? Seve Qo, SS 
: shite wl dhe D, Fete fy 


‘pide taeda. of: eoideiver. es 
_aabanois be. ite, Rewer wee ag, 


, Het eRe Fe abeebat geeiy go.) aint aga ak LY 
ri 9b wag ak eh bs aah AG gato whe PEO BY 


hdd of oad Gud ow Say Fee 44a yh Vy 
ai otwode af aos adh 24 hee broaey tha NN 


wee eae oe 


ae ewo Rp vada #74) 


*ose2 sensed vey; Prva 


Suter oe, 
Apessie 1 ye ee 
$0 240448 


ii AG 


tic .gedt begs be ede oo ami febeeia 993 4%). 


saoF tentnes eh ban AV 


Looe tod yet 8° pew 49 sent Rfcicy) al tris 


Ao sol ag beew QF aMey Pade wy oe Gteree V4 thos ath) mogtod_os daeits Ba diss fendhg 
Lge desl hi paed papi 8! gets 


S130 _ ba oest, 768 


telepoiy tbtones bo eet sep 


testh 


crane me gets _toeu nq-ha_ cot F 
La tesones, hee ae 


. ay ats an aes 
cal ie a4 palace A. so jak . 


ee +996 
We aS. .2k. 
sant Rian we Fost #3 tah 
pies 


Higa 


tagles v2 emis. graha 


foeved man at. Se 


Casi08) sreont UL Aggy 


ieee Sabie 


toseo"purs UT pesvou Siae UT 
(A2,a7 290 40yae 40 stam 


“Cabadjen veeq //  -38ed 03 pesoden oe sausnaes 


ee eS ee ee ee 


ee ee ee 


x 
% 
od 
3 
é 
Bi 
on 
4 


116 


aes or i : Ms d cage ; Hike ree PR bea OY 


“ EU Sng: Aes es ag eae : vey 


al 2h(iejtwinee ves) // 
5U(OdS POL1S> Oq JOM 411% OM 3043 // 


* + bap spaesss nd See Sie op met “ roses THT ce cag wore tas 
*8,0my i Cooee itd © pdpseys Saey am fi Of ¢ 439° io Ti aaa 
rae ee SS oe Ht ayo sea"emg ol Ay a aid oder fg. go 
_ 8. NOE Nb APM ars gt “on if %(20s¢0) ) susmaaess " 
e 
eae batho sour Paap. 8 ons abe feo zi Bk ee yy 
Bq PIMOYS 2; "OO 9y2 30 Buj3se2S pscrdas pS f/ © 
S1N2 O23 UEIIOS Baey Bn 3ey3 Y9e5 Oud AQ ff enced ae 
“pe By. UO Pash 442USD02 IsOs By? Yuaue Aj aahs // t(2040¢) woszeq"or"peesy 
Sf 3) °Ss02 wy $1 psodes pd ${yY2 UO B¥eU By 4h. = // (30sg¢) 020003 JT 
*O824 Apess{@ JOU SpsG53s BFONI 8313 AEE VED oH di. (20sua) a it 


CBETUT at ewj 2738044 
£(CaSat 7 po™ AES a, 204G¢) JE. D 


Oyd 2 Buj3403S °3541 POSA ps ByQ 48A0 GoOT = // 38u4 
°£2049090u Se Sue, Se GOO, S14T JESSY ti 
*S, SMS POIEIIV] JO BZUNOD Hyd OF} 1E}2 14) Mt 


*spsoses pa (Le 4800 passed sey doo; op // £284 ("pesn™pd™ 307 
WSYR YITYD OF SOIQELIOA JEP 1 E4314 “i 
{90404 


*@001 1023U09 62 pasn // {900u8 


oS 316453 po eaeg 
“rhe ot 0012738234 


oe $. 
ne 


Cy aan gen a: 
(STTORRTTIE ep senes reve tie Setter et 
1204uq OG42 38) (“pest pd ,07 ao, 


*(SAs7™22038 BOS) “SuOP 8) Sj{ydI VOYyR perluawessu; By // 
IUNOD Bds4 Syd /891.qe11}eAe OQ 114M P2092 asow // 

GVO Ieyy SOIEDIPU, SoveNd|s 32; 4mM-pHds © ¥uUjr12098 // 
B2UGS “SSRVENSSS 03;Um—-p!lss 4O JOqunU |se, © // 
pe2s838 Sey se “aue passg SOY 3) Wey sUsNsas // 

3t °3SEL POSEN py ays VO Busic,28IP nut // 

Oud SUj#IU;ew 34 “Spsedas pu 8844 4O saqunu // 

Sy2 86e2s9u; O23 p|y1e> Sf} sanpscosd Bry, = // 


tSINPESETS 1pscces"poees;“308 | 


psooes"pd"001373 08 


a 


ont galnneds do? sid lanageos at suubssarg afd? Ah: 
neligiames ott sete aseed aret fovines shaq to gfate \\ 
‘Nelovel £1 $£ cbevverda ef nolserene oh) we Fo V4 a we ee 
-.. ytenttues fovines solveb feutivitel mosh (yo i Be ae MeN ea aoe 


+ 


"od ees Vie 
eerr? Tle 


Ot. 
togeg soa bg: 


post_page 


post_page: procedure (Coreadd); 
local Page; 
If Rws_in_frame (Coreadd) 
then call rws_done (Coreadd); 
else da; 
Page := Paze(Coreadd); 


lo_In_progress (Page) := false; 
If fto_read_or_write (Coreadd) = "read" 
then do; 
Coreadd(Pare) := Coreadd; 
call make_accessible (Page); 
eng; 


*Weltes outstanding := Writes_outstanding 


call thread_to_top (Coreadd)> 


end; 
call notify (Event(Page)); 


refurn; 
nd post_page; 


ee 


/ 


/ 


~~ ns 


This procedure [ts responsible for changing the 
state of page control data bases when the completion 
of an t/o operation ts observed. It is Invoked 
from Individual device control routines. 


tf there was an rws here, 
or aborting. 


zo process the completion 


This was a normal page read or write. 


Identify the page Involved In this operation. 
Turn of t/o flag. 


See If read or write completed. 


Page will be made accessible In this frame. 
Insert physical core address, turn on access. 


8TT 


Handle a write completion. 

- 1; // Maintain heuristic for find_core. 

// Make this core frame the most likely 
candidate for claiming. The usual reason that a 

write was started Is that it was a good candidate for 
claiming In the first place. If Pare has been used, 
(this Itnetudes modifled) since the Usage bit was turned 


off, ftnd_core will not claim this paze now. Otherwise, 
It will Be the very next nage claimec. 


Cause any process walting for the completion of this 
{/o operation to resume. 


++ 


start_rws 


start_rws:procedure (PDrec); 


local Coreadd; 
declare Rws_active_count arithmetic; 


Coreadd := find _core(); 
Rws_tn_ frame (Coreadd). 3s seuss 


Rws_in_progress (PDrec) :@ eine) 
lo_read_or_write (Coreadd) z= "read"; 


Coreadd (POrec) :# Coreadd; 
POrec(Coreadd) := POrees s... 


7/7 =Thts procedure Initiates the moving of a 
// madified paze from the paging device to the dteks 


// This counter ts a heuristic for limiting rws activity. 


“ff Get a page of core for the rws buffer, . 
‘47> Mark this page frame as unc) at mat le, Flag 


// also lets past_nage know what to do. 

// Hark thts pd pyeorg. as having an ews tn progress. 
“ Indicate the direction of t/o for post_page. a 
- 
// Set: ue: thts relation, so. that rws abort can © 


Ad find Coreadd. 


ff Set up thls relation, -86 that ‘ris done can find POrec. 
// Thread Prec ‘out of used list, so tt can't be claimed. 


gall pd_thread_out (P0reg)1 . OW 
call device. agai! PryactoreaddiCoreadd));. PPS Start? ‘B84. PET H she ale the read. 


r(PDrec), write® 


fea meter_d yhOfekadd 
eae tive ceunt ¢ 2; 


Meter ‘w6t tof td t 
1] Malntate ineurtattes ” ves 


Numbe et a 49d, racer s s© Number_of_freepd_records * 3; ‘// ‘indicate ehge Mt phils. od ed gresers 


modttfed_ pahireaveesty :© false; 


do while Rws_active_count > 30; 


an post any_to; 
aad’ omll 


oe ree 


//] ts bwithe protests ‘6 
“ Indicate! ee ne tpl reser atin, be same as disk 
Mt Copy. | 


47 Af there Ts a large. ‘amount of . ws activity ‘going on, 
// walt for some of It tq subs! ‘de... 
/‘/ See what has compteted. 


rws_abort 


~ nws aborts Btecedure(Page); 


// This procedure Is Invoked when a page fault 
4/71 Is oe ane 3 ehveg which has an rws In progress. 
th 


“Vpead: POrec; / ee such, cases. 1) No abort has 
ue bee erent ahr i tT One, 
AF fo’® wordt re Fea de lg i erGn ries! tone yes ig bam 
us 1 Sie ha has Se sctiaeay wait fer te, 
a ave been t rw TOPO, aad t 
‘pbrec' Por (Pare): oe oh « werd wart heer i en : a 
t® POrec focee) 3 Tt het Hh aa ngerest here. 
ed a Lee ao FT in rable! a He Ret ze of eee 
Lea late (etre), gies at ef phic eto eee : ta chaes up, and the 
avy. rf rt are over 
oe seit lagtpiree), are ahs “a No more fed rate or rwe. | ae 
hark es ‘ rt_cComplete(POrec} Rws ‘ta: eee cy" waite lise; - 
“oreedatPane) im Coreeda(FOreE eae cet sa ie a vas a home for. ee. a 
_ make acces e & at i Int 
: ‘ pane {1.aesee ist ts! naa ae Sapeee- inte S 
: : ail meter. si aheraa had ct ee : wee Boas 
oe oe: coe ‘ ev Hat Bev fa ipena is fceue 
Teed bettem (Co end ryaet 7 be. Me dent doceck tenance 1 d 
ee 1 ...¢ 0 t Om Ah .. ently used. 
“oar WET) pd the read co_bott toe pordel Pt bs yi lope a nud used. 
cy-y atte ic), in_use rec) :© 3 Ones Be. 
me a: vt eg Se « Sant 2% ct lve cette ct e : Sat ate thts feuristic, 
fon m Tae (Coreadd(PBrec)) Peale, {/, Jousgke 2, ROMS Rhe Se ae e 
7 Nu e  yree ot. racards: SA ae oe sheLr a, 4 
2 : “e 1 Number_of_free_pd_ records “t;° >> “Opp HeVe rs 4 adi ees made 
by start_rws. 
// Return to page, fault with Qws _inprogress off. 
else; // Case 2. Abort already started. page_fault wil! 
// watt for fe. 


else Abort_flag (Pdrec) := reuse; // Case 1. Abort the ews. page fault will walt for It. 


ang rws_abort; 


//. Return to page fault to elther continue 
// and restart the “fault, or walt. 


je SR eed SE ee oe ee oe eb de 


rws_done 


rws_done: procedure(Coreadd); 


// This procedure handtes the completion of 
// W/o done on behalf of read-write sequences. 
// Aborts are noticed here, as well. 


Seclare Rws_active_count arithmetic, Writes outstanding arithmetic; 


docal PDrec; 


POrec := PDrec(Coreadd); 
a eats —flag (POrec) 


“Engg to. read. or_write (Coreadd) = "read" 
then Modi fled_- from_disk (PDrec) :© trues 


/ 
else Writes_outstanding :=* Writes outstanding - 1; 


Abort Complete (PDrec) := true; 


“gall notify (Event (PDrec)); 


end; 
else do; 


oe 


lo read_or_write(Coreadd) 


° 


end; 
else da; 


Rws_active_count :* Rws_active_count - 1; 
Writes outstanding 2" Writes ~outstanding - I; 


Rws_In_ frame (Coreadd) ¢s 


Rws_In_progress (Phrec) :« false; 


In_use (POrec) := false; 


d/ flese se 


ox // there 
/ 


s= "welte!; 
call device_write(Diskaddr(PNrec),Phys_| 


// Identify the concerned pd record from the 
// fleld spectally reserved for this purpose. 
// Tf an abort was requested 
// abort the rws. 
// «Nf a read was aborted, we wit! not write, 
4/ and must re-Indicate that record differs 

/ from disk. 
//° Otherwise, maintain write count. 


/1 Indtcate that we have aborted the rws. We 
// cannot make the pase addressable because 

// (notnt of fact!) we have no way to locate 

// Pare(PDrec) in the actual implementation. 

// This. ts due to not having enouzh space to 

// save the required potnters. 

// We now cause all nrocesses who faulted on 

// this paxe during the rws to resume. They 

d/o wilt atl ots “ies page faults which made them 


aan ct be oe to: tock: 


ad. anes: 
0109 re ee Jee page. fault)’ 
eturh., a dian 


/ the pares tahta, 
“ geereese ee we 


/ and st he 


ae ad Ce ee Grey age A’ hori) rws read, r west ee i on. 
Lf lo_read_or_wrlte(Coreadd) © “read® 4 yor, Weste: completion 
then da; 


ta ¢ 
If a read finished, tart the write. 


/ tndicate I/o direction for next time. 
oreadd(Coreadd)); // Start the write. 


// The write, and hence the rws, has finished, successfully 


// (l.e., without an abort. 


// Turn off rws indicator. 


// This record Is now free. 


Ter 


// Maintain the rws activity heurtstic. 
// Maintain find_core's heuristic. 


// Turn off rws Indicator for pd record. 


call pd_thread_to_top (PNrec); // This record Is now clalimable. 


Devadd (UPare(Phrec)) := Diskaddr(PDrec); // The page that was on thls pd record 
// \s now on disk. 
On_pd(Page(PfNrec)) := false; // Let all know. 
Pare (Coreadd) :# !null’s // The core block used as an rws buffer Is now 
// Immediately clalmable. 
a thread_to_top (Coreadd); // Make tt best candidate for claiming. 


3 


end rws_done; 


(ai 


aux! tiartes , Bs 


/* Although some of these short routines mirht better be expressed In 
lIne, they are conceptually modutes tn thelr own right, and 
may be called from other points In the system.*/ 


/e Small auxillary routines */ 


device_read:procedure (Devadd, Phys_Coreadd); // Calted to tInittlate a read - 
// selects correct I/o routine. 


declare Phys_Coreadd arithmetic; 


Lf Device (Devadd) “= "drum" // meter disk reads 
+ Shen call meter_disk (Devadd, “read"); // meter it 


gall (select _lo_routine_entry(Device(Devadd), “read")) 
(Phys_Devadd (Oevadd), Phys_Coreadd); // call right routine. 


end device_read; 


device_write: procedure (Devadd, Phys_Coreadd); // Called to write a page. 


rltes_outstanding :* Writes_outstanding +1; 
Gall (select_io_routine_entry (Device(Nevadd), "write™)) 
{Phys_Devadd (Nevadd), Phys_Coreadd); 


rai 


end device_write; 


_ make_accessible: procedure( Page); // Called to make @ page accessibie. 


Phys_Coreadd (Descriptor(Paze)) := Phys _Coreadd(Coreadd(Page)); // F111 In physical address. 


Addressable (Descriptor (Page)) := true: / make page addressable. 
end make_accessible; 


make_si@hacces$ibTes procedure (Page); /{ Catted to make a page non-accessible. 
Addreé¥éble (Descriptor cineca 2° false; dd wakes page: nOn-addressable. 
call clear aspociativeume 


. PY; 0. -a% o07.. flush deserlptor from associative memories. 
end make_ nonaccess 1 bT@) arts Seer ae ee tate 


meter_disk:procedute (Devadd, Type); 


ed eeng Pringtpal ft I U ° 
‘Type - exbobtmedéc setive . . ringipa procedure. 9 RY: meter ne experiment 


if not Experiment_active then return; // cannot secumu Tate data Hbutter not. wired. 
Trace_ datum: Scongtruct Trace_datum(Devadd:Nevadd, TyperType); i 


enqueue (Trace_datum, Trace_queue); 
end meter_disk; 


post_any_lo:procedure; // This routine Is called In any situation where page 
(ea // control discovers some I/o bottleneck. 
// It polls T/o routines for complete status. They wit] 
// call post_pase If any status arrives. 


deciare to_devices set literal; 
local Device literal; 


do Device := range lo_devices; // loop over all lo devices. 
call] (select_lo_routine_entry (Device,"post"™)) 
©: // make an appropriate call. 
end; 


end post_any_lo; 


vra 


_flxed_head_contro! 


/* A Typical Paging 1/0 Control Routine */ 


/* This routine Is the !/o contro! routine for the fixed-head disk. There exist routines 
almost Identical to It for the moving-head disk and drum. The routine seltect_io_routineentry 
(not glven here) ts used to select the appropriate entry of the appropriate routine ; 
given the device Identifier and the function to be performed. */ 


declare Phys_Devadd arithmetic, Phys_Coreadd arithmetic: 


fixed_head_read:procedure (Phys _Pevadd, Phys_Coreadd); // Read entry. 
call thred-head_ start (Phys_Devadd, Phys_Coreadd, read"); // Gail common quevetng routine. 
2 
fixed head_writesprocedure (Phys_Devadd, Phys_Coreadd); // Write entry. 
call fixed_head_start (Phys_Devadd, Phys_Coreadd, “write"); // Call common queueing routine. 
3 
re 
ftxed_head_start: procedure (Phys_Devadd, Phys_Coreadd, Direction); // Common routine to queue fixed-head & 
// disk requests. 
Seclace Olrection literal; : 
call fixed _head_post; 7/ See tf any operations 


// have completed. 
enaueye (construct Na EConc en 
(Phys_Devadd: Phys_Devadd, : // Construct a channel program Z a 
Phys_Coreadd: Phys_Coreadd, // and enqueue it. 
Directlon:Birection, , 
Next:"null™), Fixed_Head_Channel_Queve); 
Af (fixed head disk fs not busy) then call start_io(Fixed_Head_Channel_Nueue); 
ge oe . // There 1s now work for the 
// fixed head disk. Start It If 
// tte ts Idle. 


; 7 z care } : emda og ij an . 
PO Pow bad OL pe eS . Re we me a uer a 
flxed_head_post: pr Pp COT etc ys gees boos Los SPP Used to post completed operetlons. 4 
do lo_¥fatus :* range. (any cteaplete d/o: status); //Look' et a1) new status. <* tale 
" . 4 


oe 7 slocstegus, from fast of complete status); \\ If) Nake 10! eat of” Nerdware’ queue.” | 
“SENT bostapeae -4e cadd (lo_status)); \\ oP aw form page control. See the 
(/ Sectaration of to_status. 7 + 
_.femove lo_program (lo_status) from Fixed_Head_Chahnetquedey 9 89 E ee 
ender sean roe yae ee va AFR UR Get a gel bg ‘ 
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Appendix B 


Implementation of the Hardcore Meters 

In ‘this appendix, we relate the exact identitiés“of the measured 
events of Multics page control with which this experiment was concerned. 
This is necessary both to provide validity for what we have done, and to 
help others design similar techniques for other systems. It is assumed 
that the first appendix has been at least partially understood, perhaps 
with the overview sections fully understdod. ~ 

We also discuss here the techniques used “in implementing the Multics 
Supervisor interface for this experiment. ~*~ <""_ 

‘As should be clear from Chapter 2, we are interested in metering 
movement of pages in and out of the composite entity of core-drum. This 
"movement" in fact consists of copy creation and copy destruction. Move- 
ment “into" core drum consists of the creation of a page copy’ in core- 
drum where there previously was none, and movement "out of" core drum con- 
sists of the destruction of core or drum copies of a page, such that there 
is no copy in core-drum. - We speak of this creation and deletion as move- 
ment because it is represented as movement of pages in an LRU stack. 

We will now analyze the different types of motion in and out of 
core-drum. Pages come into core-drum éither from the outside, i.e., disk, 
or by being created in core. Pages entering from disk can only do so as 
the result of a page fault to disk or a pre*paging from disk, so a call to 
"meter_disk" (see Appendix A) was installed in the i/o dispatching 
routine to record all reads from disk. Pages created in core never 
involve input/output. For the most part, these are pages which were 


never touched before, and would thus cause a page fault no matter how 
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large core-drum were. These page faults, however, involve net thiod 
multiprogramming, i/o, nor idle time, and are thus of. slightly less 
interest in performance predictions than more general page faults. _ We. 
chose to ignore them. There is one other type of inward motion, which 
will be motivated. in our discussion of outward motion. . sian 

= Outward motion consists of oustings from core-drum, _ This. consists 
of oustings from drum (which, as can be verified: from "get_ free pd record” 
. in the last appendix can only happen if there, ig, no, copy, in core). or from 
core. . Oustings from core are only oustings from core-drum Af page con- 
trol (specifically, "“try_to_write, page”) decides that, it should not be, 
written tq the drum because of either leck. pf. apace. there, qr, the copcerned 
page. is one of ‘the special-cesed Ngtpd”, pages... We. will first sonaider .. 
the oustings from drum. The ousting of a page which is different than 
its disk copy, if one was ever made, is accomplished by. the initiation of 
a read-write sequence. (rws). These Tws initiations were thus metered as 
outward movement. The ousting of a Page which is identical to a disk 
copy is done by simply claiming the drum frame (see. "get_ free. __pd_record”), 
and this event was likewise noted. Oustings from core interest us when- 
ever they are not oustings to the drum. (We define an ousting, "from core 
to the drum" to be an ousting from core when a copy of the concerned page 
is on drum. Note that this implies an ordering of the hierarchical 
memory system.) These oustings. from core normally happen only for the . 
special "Global transparent paging device. (gtpd)" pages of the root direc- 
tory, whose treatment was already fully covered, and. in bad cases of page 


faults or rws. initiations, when there are no free. drum frames available. 
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ite Case was also covered: As an eereeees consequence of this | ‘defini- 
tion of an “ousting not to the drum, we observed a large umber Cepproxi- 
mately 2400 per Nees) of euet ines of pages | which were Yall zeros, and \ were 
all zeros ee brought into core the conjunction of ‘these geaeauents es- 
seatiaity implies that these Pages had no copies « on | ether drum or disk). * 
Some epecte® etuerimenee designed to discover the | source of this peculiar 


traffic were essentially fruitless. ‘The date reduction Programs described 


age fs 
my Bye Sosy F Seer 3 - 


in section 2.3 were modified - to fooove these ancmnloae. ousctnaey. 

One consequence of metering read-write sequence initiation as out- 
ward motion is that the aborting, or reversal due to a page fault, of a 
read-write sequence must be metered as inward motion. This was done (see 
“rws_aborc"). 

One remaining event which had to be metered was that of page destruc- 
tion. The event we chose to represent this destruction was the handing 
back of the disk frame, if one existed, to the free disk pool, of any disk 
frame at all. This happens in two cases. First, explicit page destruc- 
tion via the deletion of segments of the virtual memory requested by super- 
visor call, or their explicitly requested truncation causes this to hap- 
pen. Secondly, as we have described, find_core deallocates both disk and 
drum frames when a page containing all zeros (a void page) is found withits 
used bit off. As described in section 2.2, we are interested only in the 
destruction of pages which are not in core-drum. The destruction of any 
such non-void page will always involve the deallocation of a disk frame, 


and thus will be properly metered. The destruction of void pages is not 


*Even though this constituted about one quarter of all core-drum oustings, 
they bear absolutely no significance to the experiment. 


130 


a significant event, as they do not occupy a place in either the LRU or 
LRU extension stacks of section 2.1. The discovery of a newly void page 
by find_core also causes such an event to be recorded in the trace data 

as a page deletion. However, this page cannot be in the extension stack, 
because it was found by find_core because it was, in fact, in core. The 
data reduction programs were aware of these out-of-list deletions, and 
duly ignored them. The destruction of pages in core-drum which were never 


ousted is handled and ignored by this same mechanism. 
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Interface = 

The Multics hardcore interface for this cexperinent was designed to be 
a Sad <pecuanane part of the Multics system, and tas have as little ef- 
fect as possible on it ‘tien not oe use. Thus, a page of the virtual memory 


was allocated toe the cireular buffer described in section 2. " and its 


Aves am 


auxiliary data: When the experinent was enabled, via highly privileged 
supervisor primitive, this page was “given a dedicated | page frame and wether 
drawn “EcOul the pool of pageable core. 3 This ‘was necessary to insure that 
page éanerol: when storing date in this buffer, sould not take a page 
fault. Page control was also made to check a switch (the “enabled/dis- 
abled" switch) as to whether or not “this bad been done before atteiptine 
to reference the buffer. Another highly privileged supervisor pee 
freed the page “frame given to this buffer, resetting this switch batoce 
doing so. 

The copying of data out of this buffer, ine privileged supervisor ; 
entry point, ostensibly requires simply copying its contents into a user- 
epecreiee area. However, it was an = of ‘the interface 4 design to tusure 
that the buffer would not change wattle. the information was being copied. 
This could peppen by either the PEocessor not dding Ene > copying taking a 
page fault, or the processor, doing the copying taking a page fault refer- 
encing the user's area. Hence, to insure ‘that no ee control activity 
took place wntie this data was Deine copied, ‘the ‘aata-gathering primitive 
had to lock the ' ‘page table lock" while doing this copying. This, in 
essence, prevents page fanite on being processed, and Sane be done 


until any other process has unlocked this lock. ‘The effect of this 7ock 


is to insure that onig one processor is in page éoutral at a time. When 


Cee re ee aa ME tee ee 
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one has the page table lock locked, one must not eats a page Saar or infi- 


&. bp at, 


nite btn will result when that spharss tries to lock the page ‘table 


a tot ek : 2", aa 


. lock. to process it. “What is more, the page fault handler ‘te not recur- 


aM owrid $a te 


sivas. Ths ; it was necessary to Metre" a dedicated Page ‘of the virtual 


memory “(allocate « a a dedicated core ‘page frane- and withdrew the ‘latter from 
rede Pt oget tt oes fais ite 
the pool of pageable core). to. copy the: wired buffer into. Both Pages: 


Ho gad  .oueb 


being vived (the buffer and the temporary © copy page) ensured that no "page 


aya 
s gh e. 


fault would take place during the snes The contents. of the copy page 


a fede rt 


could then be copied to the ‘user-epectied a ni ‘after the page | table ‘eck 
BRR ag ne seb RYAkGde mee , 3s 


had been unlocked. 


A further atesiouity arose , because ‘the sequent containing & he tem- 


i ir 
3 re rere 4 ist: ad eer i o¢hse cas i 


porary copy page 2. a ‘one-per-systen segnent, ton — could not be used 


Team oe Yeh hee, 


‘by two Processes eimultaneously. Thus, a lock had ‘to “he used és exclude 


t Fou vis BGS ye OS 


such use of this. segment. “this ‘lock would be locked by any grocess wanting 


to gather data pafore it uizee the sam aiareee begs page, and unlocked attes 
ish to gal 


it had been unvired. a Process or finding the lock locked would be miti- 


eh Pons oes fast? 2 Sat ge aT Se 


programed, and the associated ‘process: notified when the Tock | was “unlocked. 


oe 3h ,teavewol 
The code witch copies the wired buffer = ne ‘temporarily ated < 


temporary cory page is ‘entered only when the ‘latter ‘has ‘been. wired. How: 


ever, it is possible | that the former may not be ive. specifically, if 
the experinent | has not ‘been ‘enabled. if thie aa pe a fatal page 
fault with the | page “table lock locked wcuta cele To ‘avoid th this, the 
enabled/ disabled svitch mst ‘be checked ‘by ent code, 1 but tt cannot check 
this switch until the | page table lock is “ectuatly Locked. only awa ie 
locked can no page possibly be aale wareferenceable, o ap othe | process 


veg 


can be in . Page control. The enabled/disabled ewitch is turned | to disabled 
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BEFORE the buffer is unwired, and the. buffer.¢can only be. made. unreference- 
able AFTER the page table lock is locked ~ AFTER ‘At had’ been wired. Thus, 
the sequence ‘of events in a call to gather ig. as follows: 


1. Attempt to lock the copy page lock; multiprogram and retry if 
_ failure. 
2. Wire the temporary copy page. 
3. Attempt to lock the page table lock; “loop until successful. 
4, Inspect the ‘enabled/disabled switeh;.copy:the wired buffer into . 
_ .the copy page if.enabled, else copy.zergs, 
5. Unlock the page table lock. a, 


6. Copy the temporary copy page out to the user-specified area. 
7. Zero the temporary copy page, and unwire it. 
8. Unlock the copy page lock; notify any waiting processes. 


The step of zeroing the copy page is done so that this page will be imme- 
diatedly claimable to find_core. This is done as both a friendly gesture 
and an attempt to keep this page off of the drum and out of the disk traf- 
fic visible to the experiment. The page frame is always void when unwired 
(returned to the pool of pageable core). 

The sequence for enabling the experiment is as follows: 


1. Wire the buffer page. 
2. Set the enabled/disabled switch to enabled. 


The sequence for disabling the experiment is as follows: 


1. Set the enabled/disabled switch to disabled. 
2. Unwire the buffer page. 


The only remaining question of locking is that of the buffer be- 
coming unwired as page control is placing data in it. This cannot happen. 
_Any page control operation sequence other than those just described can 


be summarized as: 
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Attempt to lock the page table lock; loop until successful. 


- Do all nature of page control. 


Conditionally, unlock the page table lock and end this sequence. 


SP W pe & 
e 


Check the enabled/disabled switch; add data to the buffer if and 
only if it is enabled. 


5. Go back to step 2. 

The making unreferenceable of pages by find. dere falls under step 2 
above. During the checking of the switch and the placing of data in the 
buffer, this making unreferenceable cannot happen on this processor. The 


page table lock excludes any other processor, and there is no problem. 
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Appendix C 


System Performance Graphs during Experiments 

We present here graphs of user load, cpu utilization, and paging over- 
head as functions of time of day on the” ‘dase of the. ‘two experiments >» “dtm 
21" and "dtm 23", ‘Phis data was condensed from ae ‘geaghical presentation 
of these parameters routinely prepared by the M. I F. Information Processing 
Center. It is gived here to provide a f féeling for thei velative user load 
during the experiment, and to allow a rough approximation to toal system 
headway during the axperinent to be. epiputed. This may be computed by 
multiplying the time pf “the experiment (roughly, - hours, er 50,000 seconds) 
by the fraction of the system which was not idle. ete or paging overhead 
(quite roughly, 40%) » » obtaining 20,000 seconds, and multiplying by the 
system memory reference ‘rate (400, 000 references per second); obtaining 


8 x 10° virtual memory r¢ferences. eee 
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